-- Logs begin at Mon 2022-09-26 00:56:54 UTC, end at Mon 2022-09-26 01:53:28 UTC. -- Sep 26 01:21:36.248867 dv-cinder-859203-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 26 01:21:36.256122 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 26 01:21:36.258748 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: *** Starting uWSGI 2.0.18-debian (64bit) on [Mon Sep 26 01:21:36 2022] *** Sep 26 01:21:36.258748 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 26 01:21:36.258748 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 26 01:21:36.258748 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: nodename: dv-cinder-859203-2-iscsi Sep 26 01:21:36.258748 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: machine: x86_64 Sep 26 01:21:36.258748 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: clock source: unix Sep 26 01:21:36.258748 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: pcre jit disabled Sep 26 01:21:36.258748 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: detected number of CPU cores: 4 Sep 26 01:21:36.259554 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: current working directory: / Sep 26 01:21:36.259554 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: detected binary path: /usr/bin/uwsgi-core Sep 26 01:21:36.259554 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: your processes number limit is 47680 Sep 26 01:21:36.259554 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: your memory page size is 4096 bytes Sep 26 01:21:36.259554 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: detected max file descriptor number: 2048 Sep 26 01:21:36.259554 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: lock engine: pthread robust mutexes Sep 26 01:21:36.259554 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: thunder lock: enabled Sep 26 01:21:36.259554 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 26 01:21:36.259554 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 26 01:21:36.300118 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: Python main interpreter initialized at 0x55c0633299c0 Sep 26 01:21:36.300118 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: python threads support enabled Sep 26 01:21:36.300118 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: your server socket listen backlog is limited to 100 connections Sep 26 01:21:36.300118 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: your mercy for graceful operations on workers is 90 seconds Sep 26 01:21:36.300519 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: mapped 403077 bytes (393 KB) for 2 cores Sep 26 01:21:36.300519 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: *** Operational MODE: preforking *** Sep 26 01:21:36.300817 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: *** uWSGI is running in multiple interpreter mode *** Sep 26 01:21:36.300817 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: spawned uWSGI master process (pid: 81410) Sep 26 01:21:36.301018 dv-cinder-859203-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 26 01:21:36.304389 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: spawned uWSGI worker 1 (pid: 81422, cores: 1) Sep 26 01:21:36.304389 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: spawned uWSGI worker 2 (pid: 81423, cores: 1) Sep 26 01:21:36.304389 dv-cinder-859203-2-iscsi devstack@keystone.service[81410]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 26 01:21:40.629573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 26 01:21:40.629573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: from cryptography.utils import int_from_bytes Sep 26 01:21:40.630375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 26 01:21:40.630375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: from cryptography.utils import int_from_bytes Sep 26 01:21:40.631206 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 26 01:21:40.631206 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: from cryptography.utils import int_from_bytes Sep 26 01:21:40.631660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 26 01:21:40.631660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: from cryptography.utils import int_from_bytes Sep 26 01:21:40.681806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 26 01:21:40.681987 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 26 01:21:40.682177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] command line args: [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 26 01:21:40.682447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 26 01:21:40.682624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ================================================================================ {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 26 01:21:40.682883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] admin_token = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.683123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] config_dir = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.683308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.683490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] config_source = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.683669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] debug = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.683787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 26 01:21:40.683867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.684111 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 26 01:21:40.684182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] command line args: [] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 26 01:21:40.684245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.684401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.684482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 26 01:21:40.684631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ================================================================================ {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 26 01:21:40.684720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.684891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.685017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] admin_token = **** {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.685122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.685226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] config_dir = [] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.685332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_config_append = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.685435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.685538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.685634 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] config_source = [] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.685747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_dir = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.685846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] debug = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.685959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_file = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.686063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.686172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.686298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.686298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.686495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.686571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.686749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.686875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.686985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] list_limit = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.687063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.687195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] log_config_append = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.687272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.687398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.687474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.687592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] log_dir = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.687675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.687799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] log_file = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.687875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.687999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.688076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.688197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.688283 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.688394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.688525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.688639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.688770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.688856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.688994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.689106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.689238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.689351 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.689488 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.689563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.689714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.689953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.690039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.690155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] publish_errors = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.690249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.690389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.690473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.690620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.690698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.690814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.690901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.691032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.691161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.691267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.691506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] standard_threads = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.691631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.691721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.691821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] publish_errors = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.691969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.692060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.692138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.692283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.692381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] use_journal = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.692502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.692593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] use_json = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.692672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] use_stderr = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.692748 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.692820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] use_syslog = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.692985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.693072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.693227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] standard_threads = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.693308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.693463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.693548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.693712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.693836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.693920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.693999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.694070 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] use_journal = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.694225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] use_json = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.694304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.694405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] use_stderr = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.694500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.694580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] use_syslog = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.694701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.694810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 26 01:21:40.694940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.695075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.695188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.695319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.695458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.695624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.695702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.695870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.696071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.696158 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.696326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.696505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.696588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.696670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.696824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.696911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.697074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.697159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.697313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.697393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.697568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.697661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.697810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.697894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.698053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.698131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.698289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.698370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.698540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.698617 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.698794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.698873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.699021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.699094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.699280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.699365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.699512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.699599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.699824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.699977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.700056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.700213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.700299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.700484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.700565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.700734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.700820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.700964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.701050 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.701195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.701280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.701441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.701526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.701682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.701767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.701916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.702242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.702418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.702496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.702650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.702737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.702886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.702965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.703121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.703198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.703350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.703440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.703583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.703662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.703815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.703894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.704047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.704133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.704282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.704404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.704556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.704636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.704789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.704946 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.705021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.705090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.705246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.705325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.705481 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.705566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.705714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.705797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.705953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.706030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.706183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.706268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.706422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.706507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.706644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.706719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.706883 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.707001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.707119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.707224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.707355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.707462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.707575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.707674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.707820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.707902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.708055 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.708140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.708277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.708388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.708536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.708632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.708776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.708863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.709019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.709113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.709250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.709324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.709488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.709568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.709724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.709804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.709964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.710052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.710202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.710338 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.710450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.710535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.710694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.710779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.710929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.711246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.711246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.711458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.711579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.711661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.711722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.711897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.712026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.712144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.712232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.712411 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.712540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.712660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.712742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.712899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.712991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.713139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.713522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.713616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.713690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.713813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.713902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.714054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.714135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.714292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.714371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.714529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.714617 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.714767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.714848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.715012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.715090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.715255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.715349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.715496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.715582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.715736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.715815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.715976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.716063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.716218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.716299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.716480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.716562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.716717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.716874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.716950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.717021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.717177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.717258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.717410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.717499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.717641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.717722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.717877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.717960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.718108 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.718189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.718358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.718446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.718567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.718644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.718801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.718881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.719037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.719118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.719284 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.719364 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.719527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.719614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.719768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.719857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.720000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.external = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.720080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.720232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.720337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.720538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.720620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.720775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.720855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.721006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.password = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.721086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] auth.external = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.721237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.token = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.721317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.721482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.721570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.721718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.721799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.721965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.722090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] auth.password = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.722199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.722283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] auth.token = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.722442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.722523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.722683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.722764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.722912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.723034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.723135 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.723217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.723382 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.723462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.723630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.723712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.723864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.723942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.724105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] credential.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.724185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.724396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.724481 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.724603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.724684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.724847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.724927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.725099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.725181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.725333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.725413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.725572 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.725651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.725824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.725905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.726082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.726161 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.726322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.726402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.726570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.727344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.727427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.727561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.727641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.727761 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.727932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.728059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.728179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.728314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.728451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.728602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.728696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.728846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] federation.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.728927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.729091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.729171 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.729346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.729428 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.729602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.729682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.729840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.729920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.730078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.730159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.730325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.730406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.730554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.730638 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.730796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.730879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.731041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.731129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.731278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.731359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.731514 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.731592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.731745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.731826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.731981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.732061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.732217 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.732299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.732489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.732571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.732729 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.732808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.732962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.733043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.733193 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.733273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.733424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.733505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.733668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.733751 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.733913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.733994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.734149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.734234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.734388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.734470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.734626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.734715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.734878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.734959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.735118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.735205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.735357 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.735438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.735588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.735668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.735827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.735908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.736058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.736139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.736293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.736400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.736557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.736641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.736796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.736877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.737043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.737127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.737275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.737363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.737513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.737593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.737757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.737837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.737992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.738075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.738237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.738317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.738471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.738559 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.738701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.738780 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.738938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.739018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.739168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.739257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.739406 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.739487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.739642 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.739723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.739885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.739972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.740121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.740201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.740367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.740449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.740616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.740706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.740857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.740969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.741103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.741204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.741348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.741437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.741591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.741672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.741846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.741927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.742100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.742189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.742896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.742990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.743229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.743308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.743548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.743641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.743794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.743874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.744055 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.744135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.744290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.744400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.744575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.744656 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.744821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.744900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.745067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.745156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.745303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.745389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.745536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.745617 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.745772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.745860 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.746008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.746088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.746245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.746325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.746475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.746563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.746721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.746802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.746959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.747038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.747201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.747288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.747444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.747525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.747682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.747761 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.747910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.747997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.748156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.748237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.748415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.748500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.748653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.748747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.748890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.748971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.749123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.749204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.749368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.749457 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.749613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.749693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.749846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.749925 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.750084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.750172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.750316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.750396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.750559 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.750639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.750795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.750884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.751029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.751110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.751265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.751346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.751509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.751599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.751747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.751827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.751978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.752058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.752214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.752303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.752478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.752558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.752727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.752807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.752973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.753067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.753206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.753286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.753445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.753524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.753692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] resource.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.753781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.753927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.754006 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.754173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.754254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.754409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.754491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.754647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.754727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.754881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] role.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.754962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.755130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] role.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.755214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.755361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.755442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.755610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.755690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.755854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.755934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.756094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] role.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.756174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.756353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.756483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] role.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.756597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.756687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.756840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.756921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.757086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.757168 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.757332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.757459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.757563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.757645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.757808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.758067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.758165 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.758245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.758342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.758422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.758595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.758676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.758833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.759007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.759088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.759244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.759324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.759493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.759582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.759743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.759824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.759991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.760189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.760269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.760453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.760537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.760711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.760792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.760963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.761055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.761203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.761285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.761434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.761514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.761665 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.761754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.761896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.761977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.762146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.762227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.762399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.762488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.762638 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.762719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.762871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.762950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.763105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.763193 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.763336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.763417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.763570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.763650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.763806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.763895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.764052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.764133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.764286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.764430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.764551 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.764644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.764794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] token.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.764876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.765029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.765108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.765274 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.765366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.765513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.765592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.765756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.765836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.765992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.766081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.766229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.766309 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.766460 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.766541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.766704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.766793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.766953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.767033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.767191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.767279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.767431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.767512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.767670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.767752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.767919 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.768000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.connection = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.768166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.768246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.768433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.768519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.768672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.768763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.768912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.768993 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.769144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.769232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.connection = **** {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.769379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.769461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.769613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.769694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.769847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.769928 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.770090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.770172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.770323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.770404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.770565 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.770646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.770798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.770880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.771038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.771118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.771270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.771349 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.771512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.771595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.771742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.771824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.771980 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.772061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.772217 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.772300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.772408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 26 01:21:40.772490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.772730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.772968 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.773212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.773450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 26 01:21:40.773648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81422) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 26 01:21:40.802094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.802321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.802553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.803171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.803410 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.803640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.810955 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.812403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.839969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.840202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.841886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.842127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.893363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.894175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.941951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.942184 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.942846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:40.943085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.099118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.099304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.099387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.099464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.099565 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.099645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.099759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.099845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.099975 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.100069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.100207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.100293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.123570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.123801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.123915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.124018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.124155 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.124241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.124416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.124503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.124650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.124736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.124875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.124958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.125096 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.125316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.136549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.137137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.149172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.149353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.149576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.149672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.149793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.149879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.150005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.150089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.150214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.150300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.150423 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.150508 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.150731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.150954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81422) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 26 01:21:41.167283 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.167490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 26 01:21:41.167754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.167840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.167971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 26 01:21:41.168070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 26 01:21:41.168269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.168375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.168543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 26 01:21:41.168626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.168803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.168888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.169057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.169139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.169324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.169587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.178540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.178820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.179085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.179277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.179361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.179570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.180484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.180984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.187120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.188204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.190008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.190295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.190575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.190854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.191138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.191286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.191434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.191563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.191689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.191815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.191940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.192076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.192205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.192331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.192486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.192628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.192769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.192901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.193031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.193300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.193581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.193855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.216037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.217526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.217661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.217816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.219006 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.219298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.222493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.222777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.223056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.223963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.224256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.224570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.229601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.229887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.230970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.231271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.235435 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.235737 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.236023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.236309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.236610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.236885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.236977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.237156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.237248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.237425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.237531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.237774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.238058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.238339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.238613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.238886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.253254 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.253535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.254782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.255071 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.257305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.258945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.260928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.261227 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.262164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.262470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.265337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.266654 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.268391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.269746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.271524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.271822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.272108 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.272416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.272697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.272957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.273061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.273259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.273577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.273871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.274152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.274436 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.288769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.289061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.289980 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.290294 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.290721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.291019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.291295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.291567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.297043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.298468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.298599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.300002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.300094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.300277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.301575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.301859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.302996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.303280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.303566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.303848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.304234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.304637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.304999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.305294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.305577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.305989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.315339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.315625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.315907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.316191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.316591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.316891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.317169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.317258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.317460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.317745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.318034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.318420 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.318718 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.319004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.333787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.335342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.337673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.338874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.340556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.341795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.342073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.343300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.343525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.343808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.344204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.344851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.345142 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.345549 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.350525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.351944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.352067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.353580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.354923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.355317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.355602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.355881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.356518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.356925 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.357216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.357499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.363572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.363861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.364155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.365326 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.365630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.365915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.371012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.371297 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.371586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.371863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.372138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.372432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.372707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.372994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.373124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81422) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 26 01:21:41.373250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.373377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.373529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.373667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.373813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.373952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.374080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.374215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.374341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.374467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.374592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.374719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.374899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.375176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.375467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.375754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.376022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.376292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.376586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81422) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 26 01:21:41.409408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55c0633299c0 pid: 81423 (default app) Sep 26 01:21:41.411743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55c0633299c0 pid: 81422 (default app) Sep 26 01:21:41.412622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3bd6c7-8db8-4b73-a374-13c6fc4a7e7d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:41.412911 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3bd6c7-8db8-4b73-a374-13c6fc4a7e7d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:41.413179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3bd6c7-8db8-4b73-a374-13c6fc4a7e7d None None] PATH_INFO: `/v3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:41.417359 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1/1] 10.222.0.25 () {50 vars in 791 bytes} [Mon Sep 26 01:21:41 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:21:48.230806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78e2e32-db86-4835-8b78-68ffad5c7ab4 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:48.231311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78e2e32-db86-4835-8b78-68ffad5c7ab4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:48.231417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78e2e32-db86-4835-8b78-68ffad5c7ab4 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:48.235620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1/2] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:21:48 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:21:48.241107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:48.241636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:48.241732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:48.313516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: INFO dbcounter [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] Registered counter for database keystone Sep 26 01:21:48.332544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG oslo_db.sqlalchemy.engines [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81423) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 26 01:21:48.338819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writer thread running {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 26 01:21:48.518891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG passlib.handlers.bcrypt [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81423) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 26 01:21:48.519270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG passlib.handlers.bcrypt [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81423) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 26 01:21:48.577918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:21:48.579456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:21:48.708782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 2/3] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:21:48 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 469 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 26 01:21:49.125198 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78e2e32-db86-4835-8b78-68ffad5c7ab4 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:49.197144 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: INFO dbcounter [None req-d78e2e32-db86-4835-8b78-68ffad5c7ab4 None None] Registered counter for database keystone Sep 26 01:21:49.218521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG oslo_db.sqlalchemy.engines [None req-d78e2e32-db86-4835-8b78-68ffad5c7ab4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81422) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 26 01:21:49.224651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writer thread running {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 26 01:21:49.344679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78e2e32-db86-4835-8b78-68ffad5c7ab4 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:49.348794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8042c61f-1abc-45d0-aee5-8420ba9460f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:49.349418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8042c61f-1abc-45d0-aee5-8420ba9460f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:49.349697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8042c61f-1abc-45d0-aee5-8420ba9460f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:49.349967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8042c61f-1abc-45d0-aee5-8420ba9460f8 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:49.358554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8042c61f-1abc-45d0-aee5-8420ba9460f8 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:49.416667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: WARNING keystone.common.rbac_enforcer.enforcer [None req-8042c61f-1abc-45d0-aee5-8420ba9460f8 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 26 01:21:49.417011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8042c61f-1abc-45d0-aee5-8420ba9460f8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:49.426773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 2/4] 10.222.0.25 () {58 vars in 1324 bytes} [Mon Sep 26 01:21:49 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 26 01:21:49.432726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:49.449886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056272ee-cde8-41a7-9b65-09aba7b2e47f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:49.454615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd376f76-0cbb-4227-8444-40f0ef972501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:49.455721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd376f76-0cbb-4227-8444-40f0ef972501 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:49.456006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd376f76-0cbb-4227-8444-40f0ef972501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:49.456280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd376f76-0cbb-4227-8444-40f0ef972501 admin admin] PATH_INFO: `/v3/projects/7124d36dd410484faf5f0034e732bac5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:49.457910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd376f76-0cbb-4227-8444-40f0ef972501 admin admin] RBAC: Authorizing `identity:get_project(project_id=7124d36dd410484faf5f0034e732bac5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:49.511922 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: WARNING keystone.common.rbac_enforcer.enforcer [None req-bd376f76-0cbb-4227-8444-40f0ef972501 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 26 01:21:49.512264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd376f76-0cbb-4227-8444-40f0ef972501 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:49.516773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 3/5] 10.222.0.25 () {56 vars in 1207 bytes} [Mon Sep 26 01:21:49 2022] GET /identity/v3/projects/7124d36dd410484faf5f0034e732bac5 => generated 344 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:21:51.628287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7d862-ad85-47b2-829f-9591a2433d0b None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:51.628859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7d862-ad85-47b2-829f-9591a2433d0b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:51.628859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7d862-ad85-47b2-829f-9591a2433d0b None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:51.633018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 3/6] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:21:51 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:21:51.639163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb54745-0b4c-4d3e-aa7d-f3e1ac203a16 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:51.639588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb54745-0b4c-4d3e-aa7d-f3e1ac203a16 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:51.639707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb54745-0b4c-4d3e-aa7d-f3e1ac203a16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:51.659608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9bb54745-0b4c-4d3e-aa7d-f3e1ac203a16 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:21:51.661177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9bb54745-0b4c-4d3e-aa7d-f3e1ac203a16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:21:51.669331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 4/7] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:21:51 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 26 01:21:51.941768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d7d862-ad85-47b2-829f-9591a2433d0b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:51.954794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d7d862-ad85-47b2-829f-9591a2433d0b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:51.958944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953772d7-9174-4cea-b149-736a1c9f23de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:51.959540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953772d7-9174-4cea-b149-736a1c9f23de admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:51.959821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953772d7-9174-4cea-b149-736a1c9f23de admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:51.960103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953772d7-9174-4cea-b149-736a1c9f23de admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:51.968441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953772d7-9174-4cea-b149-736a1c9f23de admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:51.970205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953772d7-9174-4cea-b149-736a1c9f23de admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:51.979011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 4/8] 10.222.0.25 () {58 vars in 1324 bytes} [Mon Sep 26 01:21:51 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:21:51.984316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb54745-0b4c-4d3e-aa7d-f3e1ac203a16 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:51.993383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb54745-0b4c-4d3e-aa7d-f3e1ac203a16 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:51.997671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b69d33-c3e7-4b73-a796-5a7afe34a426 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:51.998287 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b69d33-c3e7-4b73-a796-5a7afe34a426 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:51.998576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b69d33-c3e7-4b73-a796-5a7afe34a426 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:51.998908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b69d33-c3e7-4b73-a796-5a7afe34a426 admin admin] PATH_INFO: `/v3/users/d47510704d9d4183ab8e8ff6578a4a83` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:52.004585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b69d33-c3e7-4b73-a796-5a7afe34a426 admin admin] RBAC: Authorizing `identity:get_user(user_id=d47510704d9d4183ab8e8ff6578a4a83)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:52.006439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b69d33-c3e7-4b73-a796-5a7afe34a426 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:52.007975 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 5/9] 10.222.0.25 () {56 vars in 1198 bytes} [Mon Sep 26 01:21:51 2022] GET /identity/v3/users/d47510704d9d4183ab8e8ff6578a4a83 => generated 247 bytes in 24 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:21:54.232741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab59ab7-ca4b-43b7-9ccd-a221777f899d None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:54.233268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab59ab7-ca4b-43b7-9ccd-a221777f899d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:54.233268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab59ab7-ca4b-43b7-9ccd-a221777f899d None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:54.234596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 5/10] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:21:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:21:54.239799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2917f24b-5ac4-46c9-bb88-cd5a2fadb581 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:54.240229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2917f24b-5ac4-46c9-bb88-cd5a2fadb581 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:54.240365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2917f24b-5ac4-46c9-bb88-cd5a2fadb581 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:54.246934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedc06a3-b9dc-4e1d-bd83-4c3d5888b449 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:54.247222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedc06a3-b9dc-4e1d-bd83-4c3d5888b449 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:54.247498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedc06a3-b9dc-4e1d-bd83-4c3d5888b449 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:54.248964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 6/11] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:21:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:21:54.253960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9978ebef-a471-42ea-9722-1cbdf2cc062c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:54.254245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9978ebef-a471-42ea-9722-1cbdf2cc062c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:54.254521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9978ebef-a471-42ea-9722-1cbdf2cc062c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:54.259580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2917f24b-5ac4-46c9-bb88-cd5a2fadb581 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:21:54.260800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2917f24b-5ac4-46c9-bb88-cd5a2fadb581 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:21:54.295747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG passlib.handlers.bcrypt [None req-9978ebef-a471-42ea-9722-1cbdf2cc062c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81422) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 26 01:21:54.296135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG passlib.handlers.bcrypt [None req-9978ebef-a471-42ea-9722-1cbdf2cc062c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81422) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 26 01:21:54.342031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 6/12] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:21:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:21:54.345388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-9978ebef-a471-42ea-9722-1cbdf2cc062c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:21:54.346985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9978ebef-a471-42ea-9722-1cbdf2cc062c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:21:54.446222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 7/13] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:21:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:21:54.610738 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2917f24b-5ac4-46c9-bb88-cd5a2fadb581 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:54.635379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:54.709689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9978ebef-a471-42ea-9722-1cbdf2cc062c None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:54.716556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2917f24b-5ac4-46c9-bb88-cd5a2fadb581 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:54.750984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22849896-41e2-4f14-b2e9-fcf64ea79db1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:54.751627 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22849896-41e2-4f14-b2e9-fcf64ea79db1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:54.751906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22849896-41e2-4f14-b2e9-fcf64ea79db1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:54.752180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22849896-41e2-4f14-b2e9-fcf64ea79db1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:54.764282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22849896-41e2-4f14-b2e9-fcf64ea79db1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-22849896-41e2-4f14-b2e9-fcf64ea79db1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:21:54.773181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:21:54.774307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:21:54.775716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 7/14] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:21:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:21:54.780049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22849896-41e2-4f14-b2e9-fcf64ea79db1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:54.810550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9978ebef-a471-42ea-9722-1cbdf2cc062c None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:54.813641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:54.853100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509cdf2b-81ce-453b-9fc9-cbe65d472bd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:54.853967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509cdf2b-81ce-453b-9fc9-cbe65d472bd0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:54.854254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509cdf2b-81ce-453b-9fc9-cbe65d472bd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:54.854533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509cdf2b-81ce-453b-9fc9-cbe65d472bd0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:54.862172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:54.880084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22849896-41e2-4f14-b2e9-fcf64ea79db1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:54.914032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699c2732-0782-4c2e-a109-8257a7b6b9db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:54.914664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699c2732-0782-4c2e-a109-8257a7b6b9db None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:54.915002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699c2732-0782-4c2e-a109-8257a7b6b9db None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:54.915278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699c2732-0782-4c2e-a109-8257a7b6b9db None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:54.915944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699c2732-0782-4c2e-a109-8257a7b6b9db None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:21:54.916251 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699c2732-0782-4c2e-a109-8257a7b6b9db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:54.917680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699c2732-0782-4c2e-a109-8257a7b6b9db None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:54.922626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509cdf2b-81ce-453b-9fc9-cbe65d472bd0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:54.924130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509cdf2b-81ce-453b-9fc9-cbe65d472bd0 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:54.929193 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 8/15] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:21:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:21:54.935043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699c2732-0782-4c2e-a109-8257a7b6b9db None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:55.020516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 8/16] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:21:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 26 01:21:55.026855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509cdf2b-81ce-453b-9fc9-cbe65d472bd0 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:55.029610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:55.034369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699c2732-0782-4c2e-a109-8257a7b6b9db None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:55.068553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf75a2c9-4924-410c-8b85-cd3e7a6f1375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:55.069189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf75a2c9-4924-410c-8b85-cd3e7a6f1375 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:55.069472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf75a2c9-4924-410c-8b85-cd3e7a6f1375 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:55.069747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf75a2c9-4924-410c-8b85-cd3e7a6f1375 None admin] PATH_INFO: `/v3/users/d47510704d9d4183ab8e8ff6578a4a83` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:55.071400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf75a2c9-4924-410c-8b85-cd3e7a6f1375 None admin] RBAC: Authorizing `identity:get_user(user_id=d47510704d9d4183ab8e8ff6578a4a83)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:55.073065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf75a2c9-4924-410c-8b85-cd3e7a6f1375 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:55.074647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 9/17] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:21:54 2022] GET /identity/v3/users/d47510704d9d4183ab8e8ff6578a4a83 => generated 247 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:21:55.080218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf75a2c9-4924-410c-8b85-cd3e7a6f1375 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:55.095139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:55.128106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509cdf2b-81ce-453b-9fc9-cbe65d472bd0 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:55.161649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23c4234-77ec-4b31-99e0-c3e97280f2e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:55.162497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23c4234-77ec-4b31-99e0-c3e97280f2e2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:55.162857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23c4234-77ec-4b31-99e0-c3e97280f2e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:55.163141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23c4234-77ec-4b31-99e0-c3e97280f2e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:55.177903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf75a2c9-4924-410c-8b85-cd3e7a6f1375 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:55.181933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23c4234-77ec-4b31-99e0-c3e97280f2e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:55.183403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23c4234-77ec-4b31-99e0-c3e97280f2e2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-c23c4234-77ec-4b31-99e0-c3e97280f2e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:21:55.201464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:21:55.202428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:21:55.203497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:21:55.203497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:21:55.203497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:21:55.203497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 9/18] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:21:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:21:55.208597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23c4234-77ec-4b31-99e0-c3e97280f2e2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:55.213387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae619f6-5f6f-4eda-ba23-661d600032e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:55.214016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae619f6-5f6f-4eda-ba23-661d600032e5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:55.214297 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae619f6-5f6f-4eda-ba23-661d600032e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:55.214572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae619f6-5f6f-4eda-ba23-661d600032e5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:55.215692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae619f6-5f6f-4eda-ba23-661d600032e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:55.217172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae619f6-5f6f-4eda-ba23-661d600032e5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:55.218712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 10/19] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:21:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:21:55.224278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae619f6-5f6f-4eda-ba23-661d600032e5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:55.265636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:55.305086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:55.309591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23c4234-77ec-4b31-99e0-c3e97280f2e2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:55.325707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae619f6-5f6f-4eda-ba23-661d600032e5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:55.344305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ef98a4-e160-452e-87b7-f8e3eec675e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:55.344945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ef98a4-e160-452e-87b7-f8e3eec675e2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:55.345229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ef98a4-e160-452e-87b7-f8e3eec675e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:55.345501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ef98a4-e160-452e-87b7-f8e3eec675e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:55.346141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ef98a4-e160-452e-87b7-f8e3eec675e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:21:55.346441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ef98a4-e160-452e-87b7-f8e3eec675e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:55.347824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ef98a4-e160-452e-87b7-f8e3eec675e2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:55.358535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a73a673-59be-443e-8322-8cfa8a3c002d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:55.359132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a73a673-59be-443e-8322-8cfa8a3c002d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:55.359411 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a73a673-59be-443e-8322-8cfa8a3c002d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:55.359683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a73a673-59be-443e-8322-8cfa8a3c002d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:55.360425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a73a673-59be-443e-8322-8cfa8a3c002d None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.domain.id=default, user.id=d47510704d9d4183ab8e8ff6578a4a83) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:21:55.360733 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a73a673-59be-443e-8322-8cfa8a3c002d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:55.362158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a73a673-59be-443e-8322-8cfa8a3c002d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:55.364557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 10/20] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:21:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:21:55.369468 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 11/21] 10.222.0.25 () {56 vars in 1322 bytes} [Mon Sep 26 01:21:55 2022] GET /identity/v3/role_assignments?user.id=d47510704d9d4183ab8e8ff6578a4a83&scope.domain.id=default&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 233 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:21:57.653617 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786207c1-10c1-4e7a-a7c4-e52906700641 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:57.654122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786207c1-10c1-4e7a-a7c4-e52906700641 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:57.654197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786207c1-10c1-4e7a-a7c4-e52906700641 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:57.655863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 11/22] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:21:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:21:57.660674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bea3bf-357d-4309-97f2-68edaa385553 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:57.661490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bea3bf-357d-4309-97f2-68edaa385553 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:57.661985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bea3bf-357d-4309-97f2-68edaa385553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:57.687120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-73bea3bf-357d-4309-97f2-68edaa385553 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:21:57.688702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-73bea3bf-357d-4309-97f2-68edaa385553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:21:57.782623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 12/23] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:21:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:21:58.130621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c6b534-c8a5-4e0f-aa4c-57a7561f13e3 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:58.130942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c6b534-c8a5-4e0f-aa4c-57a7561f13e3 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:58.131184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c6b534-c8a5-4e0f-aa4c-57a7561f13e3 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:58.132535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 12/24] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:21:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:21:58.138372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1255cf-88e7-40cf-abb4-85e155d392a1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:58.138657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1255cf-88e7-40cf-abb4-85e155d392a1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:58.138931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1255cf-88e7-40cf-abb4-85e155d392a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:58.160952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8d1255cf-88e7-40cf-abb4-85e155d392a1 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:21:58.162139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8d1255cf-88e7-40cf-abb4-85e155d392a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:21:58.197856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce60e9f6-3646-4fd0-9ffc-ca7dad2a0ed9 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:58.198189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce60e9f6-3646-4fd0-9ffc-ca7dad2a0ed9 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:58.198419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce60e9f6-3646-4fd0-9ffc-ca7dad2a0ed9 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:58.199902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 13/25] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:21:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:21:58.209961 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70ca47-af03-4469-96bd-dc515cdfe36a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:58.210244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70ca47-af03-4469-96bd-dc515cdfe36a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:58.210520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70ca47-af03-4469-96bd-dc515cdfe36a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:58.249323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-5b70ca47-af03-4469-96bd-dc515cdfe36a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:21:58.250530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5b70ca47-af03-4469-96bd-dc515cdfe36a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:21:58.266809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 13/26] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:21:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:21:58.364318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1255cf-88e7-40cf-abb4-85e155d392a1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:58.389657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:58.392962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 14/27] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:21:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:21:58.497874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa8bf3e-556f-4b3c-9083-fa95480b20fb None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:58.498799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa8bf3e-556f-4b3c-9083-fa95480b20fb None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:58.499564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa8bf3e-556f-4b3c-9083-fa95480b20fb None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:58.501176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 15/28] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:21:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:21:58.506480 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbbe785-a6c3-460f-8b58-5b6c43c1da5e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:58.507030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbbe785-a6c3-460f-8b58-5b6c43c1da5e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:58.507504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbbe785-a6c3-460f-8b58-5b6c43c1da5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:58.531252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1255cf-88e7-40cf-abb4-85e155d392a1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:58.542124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-cfbbe785-a6c3-460f-8b58-5b6c43c1da5e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:21:58.544067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-cfbbe785-a6c3-460f-8b58-5b6c43c1da5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:21:58.595631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee61f7-d169-416b-9cf5-bd2aa89c3302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:58.596301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee61f7-d169-416b-9cf5-bd2aa89c3302 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:58.596607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee61f7-d169-416b-9cf5-bd2aa89c3302 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:58.596888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aee61f7-d169-416b-9cf5-bd2aa89c3302 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:58.615267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aee61f7-d169-416b-9cf5-bd2aa89c3302 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-8aee61f7-d169-416b-9cf5-bd2aa89c3302 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:21:58.620353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:21:58.621195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:21:58.622169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 14/29] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:21:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:21:58.624838 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:58.628764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee61f7-d169-416b-9cf5-bd2aa89c3302 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:58.670629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 16/30] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:21:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:21:58.732439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:58.745591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbbe785-a6c3-460f-8b58-5b6c43c1da5e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:58.796148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aee61f7-d169-416b-9cf5-bd2aa89c3302 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:58.857791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d89e02-4f09-4e5c-88a9-8d8a0ec67ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:58.858636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d89e02-4f09-4e5c-88a9-8d8a0ec67ccf None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:58.858921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d89e02-4f09-4e5c-88a9-8d8a0ec67ccf None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:58.859202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d89e02-4f09-4e5c-88a9-8d8a0ec67ccf None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:58.859884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d89e02-4f09-4e5c-88a9-8d8a0ec67ccf None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:21:58.860196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d89e02-4f09-4e5c-88a9-8d8a0ec67ccf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:58.861630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d89e02-4f09-4e5c-88a9-8d8a0ec67ccf None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:58.882341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 15/31] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:21:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:21:58.888683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d89e02-4f09-4e5c-88a9-8d8a0ec67ccf None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:58.967359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbbe785-a6c3-460f-8b58-5b6c43c1da5e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:59.018777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c25c6b-e9a4-4157-bba8-9d403fe1ec6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:59.019569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c25c6b-e9a4-4157-bba8-9d403fe1ec6b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:59.019857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c25c6b-e9a4-4157-bba8-9d403fe1ec6b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:59.020140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c25c6b-e9a4-4157-bba8-9d403fe1ec6b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:59.034390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c25c6b-e9a4-4157-bba8-9d403fe1ec6b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:59.042148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c25c6b-e9a4-4157-bba8-9d403fe1ec6b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:59.062053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d89e02-4f09-4e5c-88a9-8d8a0ec67ccf None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-99c25c6b-e9a4-4157-bba8-9d403fe1ec6b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:21:59.064469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:21:59.065336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:21:59.066145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:21:59.066145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:21:59.066145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:21:59.070397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 17/32] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:21:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:21:59.071302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c25c6b-e9a4-4157-bba8-9d403fe1ec6b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:59.093907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:59.104925 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:59.105897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d684ee-b169-42ca-a5ba-bd2cc0a83d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:59.106789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d684ee-b169-42ca-a5ba-bd2cc0a83d9f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:59.107270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d684ee-b169-42ca-a5ba-bd2cc0a83d9f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:59.107745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d684ee-b169-42ca-a5ba-bd2cc0a83d9f None admin] PATH_INFO: `/v3/users/d47510704d9d4183ab8e8ff6578a4a83` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:59.109138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d684ee-b169-42ca-a5ba-bd2cc0a83d9f None admin] RBAC: Authorizing `identity:get_user(user_id=d47510704d9d4183ab8e8ff6578a4a83)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:59.110805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d684ee-b169-42ca-a5ba-bd2cc0a83d9f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:59.113070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 16/33] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:21:58 2022] GET /identity/v3/users/d47510704d9d4183ab8e8ff6578a4a83 => generated 247 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:21:59.114177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d684ee-b169-42ca-a5ba-bd2cc0a83d9f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:59.183714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c25c6b-e9a4-4157-bba8-9d403fe1ec6b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:59.223354 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27db39c-dd75-4d96-bcef-d6129af884d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:59.223965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27db39c-dd75-4d96-bcef-d6129af884d2 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:59.224245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27db39c-dd75-4d96-bcef-d6129af884d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:59.224548 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27db39c-dd75-4d96-bcef-d6129af884d2 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:59.225220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27db39c-dd75-4d96-bcef-d6129af884d2 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:59.226276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d684ee-b169-42ca-a5ba-bd2cc0a83d9f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:59.226592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27db39c-dd75-4d96-bcef-d6129af884d2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:59.268159 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc9c1fd-9fd2-4a85-a1e9-27e371c74a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:59.269246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc9c1fd-9fd2-4a85-a1e9-27e371c74a3b None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:59.269748 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc9c1fd-9fd2-4a85-a1e9-27e371c74a3b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:59.270251 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc9c1fd-9fd2-4a85-a1e9-27e371c74a3b None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:59.271264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc9c1fd-9fd2-4a85-a1e9-27e371c74a3b None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:59.272915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc9c1fd-9fd2-4a85-a1e9-27e371c74a3b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:59.359697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 18/34] 10.222.0.25 () {60 vars in 1131 bytes} [Mon Sep 26 01:21:59 2022] POST /identity/v3/roles => generated 219 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:21:59.365582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27db39c-dd75-4d96-bcef-d6129af884d2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:59.414286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 17/35] 10.222.0.25 () {60 vars in 1131 bytes} [Mon Sep 26 01:21:59 2022] POST /identity/v3/roles => generated 225 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:21:59.416400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc9c1fd-9fd2-4a85-a1e9-27e371c74a3b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:59.486663 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:59.505058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:59.546955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27db39c-dd75-4d96-bcef-d6129af884d2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:59.553214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc9c1fd-9fd2-4a85-a1e9-27e371c74a3b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:59.598739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15108410-c596-401b-abe7-24163ac0de31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:59.599840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15108410-c596-401b-abe7-24163ac0de31 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:59.600337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15108410-c596-401b-abe7-24163ac0de31 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:59.600873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15108410-c596-401b-abe7-24163ac0de31 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:59.602727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15108410-c596-401b-abe7-24163ac0de31 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:59.605066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15108410-c596-401b-abe7-24163ac0de31 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:59.607516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 18/36] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:21:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:21:59.609229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973f289f-f3be-4d3b-a222-00ded98a2433 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:59.609712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973f289f-f3be-4d3b-a222-00ded98a2433 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:59.610174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973f289f-f3be-4d3b-a222-00ded98a2433 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:59.611936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 19/37] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:21:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:21:59.612904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973f289f-f3be-4d3b-a222-00ded98a2433 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:59.654243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c55a5d6-7330-416d-a95b-0de646c4cd6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:59.654895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c55a5d6-7330-416d-a95b-0de646c4cd6c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:59.655182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c55a5d6-7330-416d-a95b-0de646c4cd6c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:59.655496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c55a5d6-7330-416d-a95b-0de646c4cd6c None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:59.656163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c55a5d6-7330-416d-a95b-0de646c4cd6c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:21:59.656577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c55a5d6-7330-416d-a95b-0de646c4cd6c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:59.658260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c55a5d6-7330-416d-a95b-0de646c4cd6c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:59.679288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 19/38] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:21:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:21:59.682262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fa7428-3389-4f7c-86cb-ba0a7e868c55 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:59.683581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fa7428-3389-4f7c-86cb-ba0a7e868c55 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:59.684184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fa7428-3389-4f7c-86cb-ba0a7e868c55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:59.705062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-c4fa7428-3389-4f7c-86cb-ba0a7e868c55 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:21:59.706951 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c4fa7428-3389-4f7c-86cb-ba0a7e868c55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:21:59.747435 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973f289f-f3be-4d3b-a222-00ded98a2433 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:59.776257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:59.787517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9150c05-7402-4f87-905b-0f6257c22e15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:59.788305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9150c05-7402-4f87-905b-0f6257c22e15 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:59.788868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9150c05-7402-4f87-905b-0f6257c22e15 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:59.789057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9150c05-7402-4f87-905b-0f6257c22e15 None admin] PATH_INFO: `/v3/domains/default/users/d47510704d9d4183ab8e8ff6578a4a83/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:59.791190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9150c05-7402-4f87-905b-0f6257c22e15 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=d47510704d9d4183ab8e8ff6578a4a83, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:59.792689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9150c05-7402-4f87-905b-0f6257c22e15 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:21:59.794300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 20/39] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:21:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:21:59.795054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fa7428-3389-4f7c-86cb-ba0a7e868c55 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:21:59.814557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 20/40] 10.222.0.25 () {58 vars in 1361 bytes} [Mon Sep 26 01:21:59 2022] PUT /identity/v3/domains/default/users/d47510704d9d4183ab8e8ff6578a4a83/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:21:59.904484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:21:59.932851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fa7428-3389-4f7c-86cb-ba0a7e868c55 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:21:59.971595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43088b64-5e65-479e-8771-3bb20c94ce39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:21:59.972228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43088b64-5e65-479e-8771-3bb20c94ce39 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:21:59.972546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43088b64-5e65-479e-8771-3bb20c94ce39 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:21:59.972879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43088b64-5e65-479e-8771-3bb20c94ce39 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:21:59.973878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43088b64-5e65-479e-8771-3bb20c94ce39 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:21:59.975430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43088b64-5e65-479e-8771-3bb20c94ce39 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:00.015304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 21/41] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:21:59 2022] POST /identity/v3/projects => generated 301 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:00.027670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56072fe-7b5c-4119-b28a-b57577ca4746 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:00.028006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56072fe-7b5c-4119-b28a-b57577ca4746 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:00.028243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56072fe-7b5c-4119-b28a-b57577ca4746 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:00.029808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 21/42] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:00.037976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb6db8c-42f5-40b7-8fe5-ffe9212e9938 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:00.037976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb6db8c-42f5-40b7-8fe5-ffe9212e9938 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:00.037976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb6db8c-42f5-40b7-8fe5-ffe9212e9938 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:00.058431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-2eb6db8c-42f5-40b7-8fe5-ffe9212e9938 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:00.059700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2eb6db8c-42f5-40b7-8fe5-ffe9212e9938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:00.111976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56072fe-7b5c-4119-b28a-b57577ca4746 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:00.144562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 22/43] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:00.335738 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56072fe-7b5c-4119-b28a-b57577ca4746 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:00.370592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f2c8f3-ae63-4db1-b39e-5339114a7065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:00.371199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f2c8f3-ae63-4db1-b39e-5339114a7065 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:00.371479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f2c8f3-ae63-4db1-b39e-5339114a7065 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:00.371757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f2c8f3-ae63-4db1-b39e-5339114a7065 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:00.372453 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f2c8f3-ae63-4db1-b39e-5339114a7065 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:00.373868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f2c8f3-ae63-4db1-b39e-5339114a7065 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:00.389021 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 22/44] 10.222.0.25 () {60 vars in 1131 bytes} [Mon Sep 26 01:22:00 2022] POST /identity/v3/roles => generated 223 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:00.424883 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb6db8c-42f5-40b7-8fe5-ffe9212e9938 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:00.494802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:00.541890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb6db8c-42f5-40b7-8fe5-ffe9212e9938 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:00.578892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8290fdc-d5fd-45d1-8db8-b50532a6fbb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:00.579532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8290fdc-d5fd-45d1-8db8-b50532a6fbb9 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:00.579812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8290fdc-d5fd-45d1-8db8-b50532a6fbb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:00.580082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8290fdc-d5fd-45d1-8db8-b50532a6fbb9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:00.581227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8290fdc-d5fd-45d1-8db8-b50532a6fbb9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:00.582678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8290fdc-d5fd-45d1-8db8-b50532a6fbb9 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:00.584439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 23/45] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:00.589977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f2c8f3-ae63-4db1-b39e-5339114a7065 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:00.638074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:00.697360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f2c8f3-ae63-4db1-b39e-5339114a7065 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:00.731610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6601d557-f1da-4634-9eec-6310b97b570a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:00.732214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6601d557-f1da-4634-9eec-6310b97b570a None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:00.732521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6601d557-f1da-4634-9eec-6310b97b570a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:00.732798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6601d557-f1da-4634-9eec-6310b97b570a None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:00.733446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6601d557-f1da-4634-9eec-6310b97b570a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:00.734895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6601d557-f1da-4634-9eec-6310b97b570a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:00.762448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 23/46] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:22:00 2022] POST /identity/v3/projects => generated 312 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:02.123651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e115fa8a-c4fa-48d8-a283-6ae474639ea4 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:02.124176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e115fa8a-c4fa-48d8-a283-6ae474639ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:02.124264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e115fa8a-c4fa-48d8-a283-6ae474639ea4 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:02.125602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 24/47] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:02.130986 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbc966d-ced8-4e64-833f-1bde2b014d12 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:02.131377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbc966d-ced8-4e64-833f-1bde2b014d12 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:02.131552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbc966d-ced8-4e64-833f-1bde2b014d12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:02.150548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-dfbc966d-ced8-4e64-833f-1bde2b014d12 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:02.152068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-dfbc966d-ced8-4e64-833f-1bde2b014d12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:02.217905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 24/48] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:02.486567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e115fa8a-c4fa-48d8-a283-6ae474639ea4 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:02.588081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e115fa8a-c4fa-48d8-a283-6ae474639ea4 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:02.613963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:02.621961 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd6ffc5-c27c-4c83-b8a2-d6937858bd5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:02.622588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd6ffc5-c27c-4c83-b8a2-d6937858bd5c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:02.622870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd6ffc5-c27c-4c83-b8a2-d6937858bd5c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:02.623145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd6ffc5-c27c-4c83-b8a2-d6937858bd5c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:02.634540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd6ffc5-c27c-4c83-b8a2-d6937858bd5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-bdd6ffc5-c27c-4c83-b8a2-d6937858bd5c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:02.640920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:02.641926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:02.642892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 25/49] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:22:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:02.647867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbc966d-ced8-4e64-833f-1bde2b014d12 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:02.673188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:02.748817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbc966d-ced8-4e64-833f-1bde2b014d12 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:02.782644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce866df-dc47-45f4-ba91-e01e460915de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:02.783257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce866df-dc47-45f4-ba91-e01e460915de None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:02.783540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce866df-dc47-45f4-ba91-e01e460915de None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:02.783819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce866df-dc47-45f4-ba91-e01e460915de None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:02.784522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce866df-dc47-45f4-ba91-e01e460915de None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:02.784835 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce866df-dc47-45f4-ba91-e01e460915de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:02.786246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce866df-dc47-45f4-ba91-e01e460915de None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:02.794705 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 25/50] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:02.800695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd6ffc5-c27c-4c83-b8a2-d6937858bd5c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:02.927667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7461569c-174f-4887-9661-621a964b89f1 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:02.927953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7461569c-174f-4887-9661-621a964b89f1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:02.928236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7461569c-174f-4887-9661-621a964b89f1 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:02.929683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 26/51] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:22:02.936205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2f9d3e-e626-4ed6-aca6-3c431bc38d31 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:02.936531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2f9d3e-e626-4ed6-aca6-3c431bc38d31 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:02.936817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2f9d3e-e626-4ed6-aca6-3c431bc38d31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:02.956889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1e2f9d3e-e626-4ed6-aca6-3c431bc38d31 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:02.958149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1e2f9d3e-e626-4ed6-aca6-3c431bc38d31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:03.012158 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd6ffc5-c27c-4c83-b8a2-d6937858bd5c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:03.027367 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 27/52] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:03.049673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3409e3fc-fb25-4a0b-a8dd-fefd448174ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:03.050756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3409e3fc-fb25-4a0b-a8dd-fefd448174ae None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:03.051382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3409e3fc-fb25-4a0b-a8dd-fefd448174ae None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:03.051470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3409e3fc-fb25-4a0b-a8dd-fefd448174ae None admin] PATH_INFO: `/v3/users/d47510704d9d4183ab8e8ff6578a4a83` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:03.052925 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3409e3fc-fb25-4a0b-a8dd-fefd448174ae None admin] RBAC: Authorizing `identity:get_user(user_id=d47510704d9d4183ab8e8ff6578a4a83)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:03.054444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3409e3fc-fb25-4a0b-a8dd-fefd448174ae None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:03.057106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 26/53] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:02 2022] GET /identity/v3/users/d47510704d9d4183ab8e8ff6578a4a83 => generated 247 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:03.062517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2f9d3e-e626-4ed6-aca6-3c431bc38d31 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:03.080372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:03.159458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2f9d3e-e626-4ed6-aca6-3c431bc38d31 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:03.193925 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff8b910-d260-422b-9225-00c5850505c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:03.194552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff8b910-d260-422b-9225-00c5850505c3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:03.194835 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff8b910-d260-422b-9225-00c5850505c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:03.195109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff8b910-d260-422b-9225-00c5850505c3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:03.196279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff8b910-d260-422b-9225-00c5850505c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:03.197892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff8b910-d260-422b-9225-00c5850505c3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:03.199442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 28/54] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:03.205657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3409e3fc-fb25-4a0b-a8dd-fefd448174ae None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:03.261234 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:03.296904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff8b910-d260-422b-9225-00c5850505c3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:03.304799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3409e3fc-fb25-4a0b-a8dd-fefd448174ae None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:03.338956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cc2747-9345-4ca6-955f-852793213d43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:03.339569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cc2747-9345-4ca6-955f-852793213d43 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:03.339853 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cc2747-9345-4ca6-955f-852793213d43 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:03.340186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cc2747-9345-4ca6-955f-852793213d43 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:03.341224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cc2747-9345-4ca6-955f-852793213d43 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.domain.id=default, user.id=d47510704d9d4183ab8e8ff6578a4a83) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:03.341659 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cc2747-9345-4ca6-955f-852793213d43 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:03.343843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cc2747-9345-4ca6-955f-852793213d43 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:03.368664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 27/55] 10.222.0.25 () {56 vars in 1322 bytes} [Mon Sep 26 01:22:03 2022] GET /identity/v3/role_assignments?user.id=d47510704d9d4183ab8e8ff6578a4a83&scope.domain.id=default&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 530 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:03.383763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:03.402310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff8b910-d260-422b-9225-00c5850505c3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:03.438990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5805fc-af02-4eba-81de-abcfd3e589ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:03.439636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5805fc-af02-4eba-81de-abcfd3e589ef None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:03.439921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5805fc-af02-4eba-81de-abcfd3e589ef None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:03.440197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5805fc-af02-4eba-81de-abcfd3e589ef None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:03.441390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5805fc-af02-4eba-81de-abcfd3e589ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:03.442862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5805fc-af02-4eba-81de-abcfd3e589ef None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:03.444491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 29/56] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:03.450349 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cc2747-9345-4ca6-955f-852793213d43 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:03.557524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cc2747-9345-4ca6-955f-852793213d43 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:03.578272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:03.592756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99bb5ed-0e0d-4756-8624-fc22431cc400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:03.593370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99bb5ed-0e0d-4756-8624-fc22431cc400 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:03.593655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99bb5ed-0e0d-4756-8624-fc22431cc400 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:03.593933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99bb5ed-0e0d-4756-8624-fc22431cc400 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:03.594602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99bb5ed-0e0d-4756-8624-fc22431cc400 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:03.596062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99bb5ed-0e0d-4756-8624-fc22431cc400 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:03.622390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 28/57] 10.222.0.25 () {60 vars in 1140 bytes} [Mon Sep 26 01:22:03 2022] POST /identity/v3/projects => generated 298 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:05.754803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705be5f1-c365-4554-bace-43ca98cd1d78 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:05.755341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705be5f1-c365-4554-bace-43ca98cd1d78 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:05.755341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705be5f1-c365-4554-bace-43ca98cd1d78 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:05.756722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 30/58] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:05.761959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee479ee-2c59-4b72-ba57-100451e66438 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:05.762368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee479ee-2c59-4b72-ba57-100451e66438 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:05.762552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee479ee-2c59-4b72-ba57-100451e66438 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:05.781023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0ee479ee-2c59-4b72-ba57-100451e66438 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:05.782231 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0ee479ee-2c59-4b72-ba57-100451e66438 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:05.847726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 29/59] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:06.119083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705be5f1-c365-4554-bace-43ca98cd1d78 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:06.222365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705be5f1-c365-4554-bace-43ca98cd1d78 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:06.256694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a79f687-0f20-40de-8cfe-2302c348aaa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:06.257327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a79f687-0f20-40de-8cfe-2302c348aaa3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:06.257611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a79f687-0f20-40de-8cfe-2302c348aaa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:06.257885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a79f687-0f20-40de-8cfe-2302c348aaa3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:06.259242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a79f687-0f20-40de-8cfe-2302c348aaa3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:06.260728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a79f687-0f20-40de-8cfe-2302c348aaa3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:06.262464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 31/60] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:06.268033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee479ee-2c59-4b72-ba57-100451e66438 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:06.366944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee479ee-2c59-4b72-ba57-100451e66438 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:06.369197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:06.402976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81219dd-3b16-43c5-9117-43ed656ddecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:06.403581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81219dd-3b16-43c5-9117-43ed656ddecd None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:06.403862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81219dd-3b16-43c5-9117-43ed656ddecd None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:06.404140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81219dd-3b16-43c5-9117-43ed656ddecd None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:06.404823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81219dd-3b16-43c5-9117-43ed656ddecd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:06.406278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81219dd-3b16-43c5-9117-43ed656ddecd None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:06.432843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 30/61] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:22:06 2022] POST /identity/v3/users => generated 275 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:09.613113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46061d78-6946-484a-8796-c101aa1cb33a None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:09.613635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46061d78-6946-484a-8796-c101aa1cb33a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:09.613708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46061d78-6946-484a-8796-c101aa1cb33a None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:09.615305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 32/62] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:09.630387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f3ce45-5b7e-4410-882c-e03ab36ef98d None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:09.630916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f3ce45-5b7e-4410-882c-e03ab36ef98d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:09.630916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f3ce45-5b7e-4410-882c-e03ab36ef98d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:09.649843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-a8f3ce45-5b7e-4410-882c-e03ab36ef98d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:09.651061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a8f3ce45-5b7e-4410-882c-e03ab36ef98d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:09.737729 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 31/63] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:09.742360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3ab027-411e-4412-b288-c718d9d87e44 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:09.742645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3ab027-411e-4412-b288-c718d9d87e44 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:09.742929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3ab027-411e-4412-b288-c718d9d87e44 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:09.745159 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 33/64] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:09.761665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcff5d0a-11c5-4c89-92bc-aaa26fb6fb30 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:09.761958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcff5d0a-11c5-4c89-92bc-aaa26fb6fb30 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:09.762237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcff5d0a-11c5-4c89-92bc-aaa26fb6fb30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:09.789408 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-dcff5d0a-11c5-4c89-92bc-aaa26fb6fb30 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:09.791165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-dcff5d0a-11c5-4c89-92bc-aaa26fb6fb30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:09.884704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 32/65] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:10.105795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3ab027-411e-4412-b288-c718d9d87e44 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:10.133590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:10.226345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcff5d0a-11c5-4c89-92bc-aaa26fb6fb30 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:10.247235 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:10.249817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3ab027-411e-4412-b288-c718d9d87e44 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:10.288211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4192f0e9-c5b9-4aab-8ace-e049688e0890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:10.288885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4192f0e9-c5b9-4aab-8ace-e049688e0890 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.289199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4192f0e9-c5b9-4aab-8ace-e049688e0890 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.289477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4192f0e9-c5b9-4aab-8ace-e049688e0890 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.301107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4192f0e9-c5b9-4aab-8ace-e049688e0890 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4192f0e9-c5b9-4aab-8ace-e049688e0890 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:10.306093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:10.307201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 34/66] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:10.313008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78c8497-17fe-42d4-92f7-e3947c9d3f48 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.313681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78c8497-17fe-42d4-92f7-e3947c9d3f48 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.313681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78c8497-17fe-42d4-92f7-e3947c9d3f48 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.313681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 35/67] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:10.313681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef86e9-30f8-420f-a806-0a36cce8d1f7 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.313681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef86e9-30f8-420f-a806-0a36cce8d1f7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.313681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef86e9-30f8-420f-a806-0a36cce8d1f7 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.313681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 36/68] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:10.317137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aef86e9-30f8-420f-a806-0a36cce8d1f7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:10.344521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcff5d0a-11c5-4c89-92bc-aaa26fb6fb30 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:10.381049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64492cac-9902-442b-ab49-0987392ff83b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:10.381906 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64492cac-9902-442b-ab49-0987392ff83b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.382368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64492cac-9902-442b-ab49-0987392ff83b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.382781 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64492cac-9902-442b-ab49-0987392ff83b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.394116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64492cac-9902-442b-ab49-0987392ff83b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-64492cac-9902-442b-ab49-0987392ff83b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:10.399264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:10.400702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:10.401467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:10.406536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 33/69] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:22:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:22:10.407032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:10.408291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992c7fb-3da6-42f5-9aac-eb4dd1c76529 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.408424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992c7fb-3da6-42f5-9aac-eb4dd1c76529 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.408759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992c7fb-3da6-42f5-9aac-eb4dd1c76529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.427222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-8992c7fb-3da6-42f5-9aac-eb4dd1c76529 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:10.428446 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8992c7fb-3da6-42f5-9aac-eb4dd1c76529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:10.429858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aef86e9-30f8-420f-a806-0a36cce8d1f7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:10.471653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1570efa5-ea6f-49fb-a175-4f52224669b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:10.472284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1570efa5-ea6f-49fb-a175-4f52224669b2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.472598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1570efa5-ea6f-49fb-a175-4f52224669b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.472872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1570efa5-ea6f-49fb-a175-4f52224669b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.473560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1570efa5-ea6f-49fb-a175-4f52224669b2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:10.473864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1570efa5-ea6f-49fb-a175-4f52224669b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:10.475319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1570efa5-ea6f-49fb-a175-4f52224669b2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:10.485256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 37/70] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:10.490965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486c0d52-ddba-461a-8046-c5335ac45ff3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.491254 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486c0d52-ddba-461a-8046-c5335ac45ff3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.491536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486c0d52-ddba-461a-8046-c5335ac45ff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.507460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 34/71] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:10.508692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8992c7fb-3da6-42f5-9aac-eb4dd1c76529 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:10.512017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-486c0d52-ddba-461a-8046-c5335ac45ff3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:10.513261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-486c0d52-ddba-461a-8046-c5335ac45ff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:10.530294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:10.599971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 38/72] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:10.608520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486c0d52-ddba-461a-8046-c5335ac45ff3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:10.633181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8992c7fb-3da6-42f5-9aac-eb4dd1c76529 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:10.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48644ea1-dcee-44e7-be37-8a914430370b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:10.683619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48644ea1-dcee-44e7-be37-8a914430370b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.684050 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48644ea1-dcee-44e7-be37-8a914430370b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.684540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48644ea1-dcee-44e7-be37-8a914430370b None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.685597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48644ea1-dcee-44e7-be37-8a914430370b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:10.686645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48644ea1-dcee-44e7-be37-8a914430370b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:10.688477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48644ea1-dcee-44e7-be37-8a914430370b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:10.689734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:10.704645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 35/73] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:10.706204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf6277b-f4c6-4e47-9c35-40a49cc06267 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.706697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf6277b-f4c6-4e47-9c35-40a49cc06267 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.707150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf6277b-f4c6-4e47-9c35-40a49cc06267 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.708556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 36/74] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:10.714109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804f0160-3e3d-4001-a69b-9a6f4e48619b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.714653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804f0160-3e3d-4001-a69b-9a6f4e48619b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.715128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804f0160-3e3d-4001-a69b-9a6f4e48619b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.726954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486c0d52-ddba-461a-8046-c5335ac45ff3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:10.736615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-804f0160-3e3d-4001-a69b-9a6f4e48619b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:10.738144 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-804f0160-3e3d-4001-a69b-9a6f4e48619b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:10.764286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e00665d-5a81-4511-b842-228a794e5f17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:10.764949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e00665d-5a81-4511-b842-228a794e5f17 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.765230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e00665d-5a81-4511-b842-228a794e5f17 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.765503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e00665d-5a81-4511-b842-228a794e5f17 None admin] PATH_INFO: `/v3/users/18674c9f5ca2436b921d8f52a40444be` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.781218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e00665d-5a81-4511-b842-228a794e5f17 None admin] RBAC: Authorizing `identity:get_user(user_id=18674c9f5ca2436b921d8f52a40444be)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:10.782868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e00665d-5a81-4511-b842-228a794e5f17 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:10.784544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 39/75] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:10 2022] GET /identity/v3/users/18674c9f5ca2436b921d8f52a40444be => generated 275 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:10.789905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e00665d-5a81-4511-b842-228a794e5f17 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:10.811633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:10.816831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 37/76] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:10.825292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804f0160-3e3d-4001-a69b-9a6f4e48619b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:10.896973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e00665d-5a81-4511-b842-228a794e5f17 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:10.926751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:10.934562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844f1f2e-d9e9-4650-8107-8f77295fae6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:10.935304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f1f2e-d9e9-4650-8107-8f77295fae6e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.935489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f1f2e-d9e9-4650-8107-8f77295fae6e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.935769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f1f2e-d9e9-4650-8107-8f77295fae6e None admin] PATH_INFO: `/v3/users/d47510704d9d4183ab8e8ff6578a4a83` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:10.936944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844f1f2e-d9e9-4650-8107-8f77295fae6e None admin] RBAC: Authorizing `identity:get_user(user_id=d47510704d9d4183ab8e8ff6578a4a83)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:10.938392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844f1f2e-d9e9-4650-8107-8f77295fae6e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:10.940129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 40/77] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:10 2022] GET /identity/v3/users/d47510704d9d4183ab8e8ff6578a4a83 => generated 247 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:10.940843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844f1f2e-d9e9-4650-8107-8f77295fae6e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:10.947691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804f0160-3e3d-4001-a69b-9a6f4e48619b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:10.984147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c17515-67e7-4212-84be-3974a5026608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:10.984835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c17515-67e7-4212-84be-3974a5026608 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:10.985120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c17515-67e7-4212-84be-3974a5026608 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:10.985397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c17515-67e7-4212-84be-3974a5026608 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.045593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844f1f2e-d9e9-4650-8107-8f77295fae6e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:11.064797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:11.067198 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c17515-67e7-4212-84be-3974a5026608 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:11.068692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c17515-67e7-4212-84be-3974a5026608 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:11.086787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496f1028-aa17-4433-ad30-de91a6ac37ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:11.088069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496f1028-aa17-4433-ad30-de91a6ac37ae None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:11.088361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496f1028-aa17-4433-ad30-de91a6ac37ae None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:11.088653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496f1028-aa17-4433-ad30-de91a6ac37ae None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.099197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496f1028-aa17-4433-ad30-de91a6ac37ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-496f1028-aa17-4433-ad30-de91a6ac37ae None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:11.104802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:11.105749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:11.106515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 41/78] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:11.107249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496f1028-aa17-4433-ad30-de91a6ac37ae None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.164476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:11.170215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 38/79] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:11.171074 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c17515-67e7-4212-84be-3974a5026608 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.210966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496f1028-aa17-4433-ad30-de91a6ac37ae None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:11.244799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85183389-2ab1-4878-9c1b-bbff74264601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:11.245426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85183389-2ab1-4878-9c1b-bbff74264601 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:11.245713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85183389-2ab1-4878-9c1b-bbff74264601 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:11.245992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85183389-2ab1-4878-9c1b-bbff74264601 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.255984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85183389-2ab1-4878-9c1b-bbff74264601 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-85183389-2ab1-4878-9c1b-bbff74264601 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:11.261527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:11.262568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:11.263608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 42/80] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:11.264523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85183389-2ab1-4878-9c1b-bbff74264601 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.269859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:11.272936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c17515-67e7-4212-84be-3974a5026608 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:11.308897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59681df2-bf11-4e8e-a319-752e8ad9076e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:11.309435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59681df2-bf11-4e8e-a319-752e8ad9076e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:11.309715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59681df2-bf11-4e8e-a319-752e8ad9076e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:11.309989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59681df2-bf11-4e8e-a319-752e8ad9076e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.365882 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85183389-2ab1-4878-9c1b-bbff74264601 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:11.377534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59681df2-bf11-4e8e-a319-752e8ad9076e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:11.379016 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59681df2-bf11-4e8e-a319-752e8ad9076e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:11.384455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:11.402502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde0424-c36c-4452-8e0e-28e1f9b54802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:11.403133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde0424-c36c-4452-8e0e-28e1f9b54802 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:11.403420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde0424-c36c-4452-8e0e-28e1f9b54802 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:11.403696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde0424-c36c-4452-8e0e-28e1f9b54802 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.404852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fde0424-c36c-4452-8e0e-28e1f9b54802 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:11.406335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fde0424-c36c-4452-8e0e-28e1f9b54802 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:11.408080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 43/81] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:11.408684 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde0424-c36c-4452-8e0e-28e1f9b54802 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.480581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:11.484674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 39/82] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:11.490182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59681df2-bf11-4e8e-a319-752e8ad9076e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.510513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde0424-c36c-4452-8e0e-28e1f9b54802 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:11.544853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e97921c-2edc-4d89-a477-5f165e6048b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:11.545459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e97921c-2edc-4d89-a477-5f165e6048b1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:11.545741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e97921c-2edc-4d89-a477-5f165e6048b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:11.546021 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e97921c-2edc-4d89-a477-5f165e6048b1 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.546724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e97921c-2edc-4d89-a477-5f165e6048b1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:11.547034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e97921c-2edc-4d89-a477-5f165e6048b1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:11.548463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e97921c-2edc-4d89-a477-5f165e6048b1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:11.557088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 44/83] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:11.557088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e97921c-2edc-4d89-a477-5f165e6048b1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.593196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59681df2-bf11-4e8e-a319-752e8ad9076e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:11.606550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:11.627655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1af470f-63bf-4a36-8e1e-40764afb09d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:11.628281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1af470f-63bf-4a36-8e1e-40764afb09d3 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:11.628594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1af470f-63bf-4a36-8e1e-40764afb09d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:11.628868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1af470f-63bf-4a36-8e1e-40764afb09d3 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.629986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1af470f-63bf-4a36-8e1e-40764afb09d3 None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:11.631407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1af470f-63bf-4a36-8e1e-40764afb09d3 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:11.633057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 40/84] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:11 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:11.633798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1af470f-63bf-4a36-8e1e-40764afb09d3 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.660418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e97921c-2edc-4d89-a477-5f165e6048b1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:11.694327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13f9ef1-124c-4222-87cf-4409f4d4f740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:11.694926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f9ef1-124c-4222-87cf-4409f4d4f740 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:11.695206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f9ef1-124c-4222-87cf-4409f4d4f740 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:11.695486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f9ef1-124c-4222-87cf-4409f4d4f740 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.696160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13f9ef1-124c-4222-87cf-4409f4d4f740 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:11.696508 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13f9ef1-124c-4222-87cf-4409f4d4f740 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:11.697902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13f9ef1-124c-4222-87cf-4409f4d4f740 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:11.701565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:11.706557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 45/85] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:11.707128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13f9ef1-124c-4222-87cf-4409f4d4f740 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.736719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1af470f-63bf-4a36-8e1e-40764afb09d3 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:11.769473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60f7501-50ae-497c-b07e-f96359a8b1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:11.770081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f7501-50ae-497c-b07e-f96359a8b1e3 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:11.770361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f7501-50ae-497c-b07e-f96359a8b1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:11.770639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60f7501-50ae-497c-b07e-f96359a8b1e3 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.771286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60f7501-50ae-497c-b07e-f96359a8b1e3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:11.772742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60f7501-50ae-497c-b07e-f96359a8b1e3 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:11.795589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 41/86] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:22:11 2022] POST /identity/v3/users => generated 289 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:11.796445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60f7501-50ae-497c-b07e-f96359a8b1e3 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.815719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13f9ef1-124c-4222-87cf-4409f4d4f740 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:11.828372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:11.857430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d006fa-3ccb-4c96-8efc-fede9024c41e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:11.857999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d006fa-3ccb-4c96-8efc-fede9024c41e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:11.858277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d006fa-3ccb-4c96-8efc-fede9024c41e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:11.858568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d006fa-3ccb-4c96-8efc-fede9024c41e None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.859853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d006fa-3ccb-4c96-8efc-fede9024c41e None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:11.861404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d006fa-3ccb-4c96-8efc-fede9024c41e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:11.863205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 46/87] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:11 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:11.863995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d006fa-3ccb-4c96-8efc-fede9024c41e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.920245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60f7501-50ae-497c-b07e-f96359a8b1e3 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:11.948393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:11.964004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066bad2b-358e-42df-8a2d-8b78af68b254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:11.964959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066bad2b-358e-42df-8a2d-8b78af68b254 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:11.965303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066bad2b-358e-42df-8a2d-8b78af68b254 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:11.965586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066bad2b-358e-42df-8a2d-8b78af68b254 None admin] PATH_INFO: `/v3/users/18674c9f5ca2436b921d8f52a40444be` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:11.966913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066bad2b-358e-42df-8a2d-8b78af68b254 None admin] RBAC: Authorizing `identity:get_user(user_id=18674c9f5ca2436b921d8f52a40444be)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:11.968410 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066bad2b-358e-42df-8a2d-8b78af68b254 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:11.970179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 42/88] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:11 2022] GET /identity/v3/users/18674c9f5ca2436b921d8f52a40444be => generated 275 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:11.971100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066bad2b-358e-42df-8a2d-8b78af68b254 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:11.980116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d006fa-3ccb-4c96-8efc-fede9024c41e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:12.021594 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef72e369-c667-4176-aebc-3f18f21d3c29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:12.022207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72e369-c667-4176-aebc-3f18f21d3c29 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:12.022494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72e369-c667-4176-aebc-3f18f21d3c29 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:12.022773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72e369-c667-4176-aebc-3f18f21d3c29 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:12.023510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef72e369-c667-4176-aebc-3f18f21d3c29 None admin] RBAC: Adding query filter params (role.id=27476172dbb84c6e86b9070e02d06100, scope.project.id=5f976380e1494db480dc80868d51a0bc, user.id=18674c9f5ca2436b921d8f52a40444be) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:12.023823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef72e369-c667-4176-aebc-3f18f21d3c29 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:12.025551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef72e369-c667-4176-aebc-3f18f21d3c29 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:12.032782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 47/89] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:11 2022] GET /identity/v3/role_assignments?user.id=18674c9f5ca2436b921d8f52a40444be&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=27476172dbb84c6e86b9070e02d06100 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:12.033598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef72e369-c667-4176-aebc-3f18f21d3c29 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:12.093837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:12.115132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066bad2b-358e-42df-8a2d-8b78af68b254 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:12.157682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0389e2f4-3be5-4ce9-a573-26df19bcc92b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:12.159191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0389e2f4-3be5-4ce9-a573-26df19bcc92b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:12.159668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0389e2f4-3be5-4ce9-a573-26df19bcc92b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:12.160301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0389e2f4-3be5-4ce9-a573-26df19bcc92b None admin] PATH_INFO: `/v3/users/18674c9f5ca2436b921d8f52a40444be` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:12.161620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0389e2f4-3be5-4ce9-a573-26df19bcc92b None admin] RBAC: Authorizing `identity:get_user(user_id=18674c9f5ca2436b921d8f52a40444be)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:12.163546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0389e2f4-3be5-4ce9-a573-26df19bcc92b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:12.166632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 43/90] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:11 2022] GET /identity/v3/users/18674c9f5ca2436b921d8f52a40444be => generated 275 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:12.167613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0389e2f4-3be5-4ce9-a573-26df19bcc92b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:12.179608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef72e369-c667-4176-aebc-3f18f21d3c29 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:12.226058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2ba8c6-31ab-47f4-9c6a-18ba35ecae5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:12.227046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ba8c6-31ab-47f4-9c6a-18ba35ecae5e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:12.228084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ba8c6-31ab-47f4-9c6a-18ba35ecae5e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:12.228915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ba8c6-31ab-47f4-9c6a-18ba35ecae5e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:12.229967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2ba8c6-31ab-47f4-9c6a-18ba35ecae5e None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=5f976380e1494db480dc80868d51a0bc, user.id=d47510704d9d4183ab8e8ff6578a4a83) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:12.230492 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2ba8c6-31ab-47f4-9c6a-18ba35ecae5e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:12.232229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2ba8c6-31ab-47f4-9c6a-18ba35ecae5e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:12.238608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 48/91] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:12 2022] GET /identity/v3/role_assignments?user.id=d47510704d9d4183ab8e8ff6578a4a83&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 259 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:12.239864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2ba8c6-31ab-47f4-9c6a-18ba35ecae5e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:12.253430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:12.318571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0389e2f4-3be5-4ce9-a573-26df19bcc92b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:12.371244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04543d2c-797d-4b5f-94ab-73659f820f15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:12.371782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04543d2c-797d-4b5f-94ab-73659f820f15 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:12.372063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04543d2c-797d-4b5f-94ab-73659f820f15 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:12.372360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04543d2c-797d-4b5f-94ab-73659f820f15 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:12.394636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2ba8c6-31ab-47f4-9c6a-18ba35ecae5e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:12.413384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:12.498028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78185d0c-1409-4f4c-9956-e6ae1e493faa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:12.498656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78185d0c-1409-4f4c-9956-e6ae1e493faa None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:12.498951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78185d0c-1409-4f4c-9956-e6ae1e493faa None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:12.499229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78185d0c-1409-4f4c-9956-e6ae1e493faa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:12.515262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04543d2c-797d-4b5f-94ab-73659f820f15 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:12.517107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04543d2c-797d-4b5f-94ab-73659f820f15 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:12.626152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:12.681559 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78185d0c-1409-4f4c-9956-e6ae1e493faa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:12.683100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78185d0c-1409-4f4c-9956-e6ae1e493faa None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:12.723208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 44/92] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 556 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:12.730066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04543d2c-797d-4b5f-94ab-73659f820f15 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:12.876861 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04543d2c-797d-4b5f-94ab-73659f820f15 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:12.921028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 49/93] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 681 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:12.931022 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78185d0c-1409-4f4c-9956-e6ae1e493faa None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:12.942804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c80ffe5-3c6d-4c70-97fd-50b07c66410d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:12.943807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c80ffe5-3c6d-4c70-97fd-50b07c66410d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:12.944295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c80ffe5-3c6d-4c70-97fd-50b07c66410d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:12.944804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c80ffe5-3c6d-4c70-97fd-50b07c66410d None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:12.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c80ffe5-3c6d-4c70-97fd-50b07c66410d None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:12.955807 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:12.956941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c80ffe5-3c6d-4c70-97fd-50b07c66410d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:12.959383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 45/94] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:12 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:12.969410 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c80ffe5-3c6d-4c70-97fd-50b07c66410d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:13.045851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:13.178878 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78185d0c-1409-4f4c-9956-e6ae1e493faa None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:13.242627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c80ffe5-3c6d-4c70-97fd-50b07c66410d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:13.262820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12809aa4-ce64-4cfd-b869-016143c6a363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:13.263777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12809aa4-ce64-4cfd-b869-016143c6a363 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:13.264200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12809aa4-ce64-4cfd-b869-016143c6a363 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:13.264639 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12809aa4-ce64-4cfd-b869-016143c6a363 None admin] PATH_INFO: `/v3/projects/057afb8734e34b9687d7b36edb0034b4` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:13.266275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12809aa4-ce64-4cfd-b869-016143c6a363 None admin] RBAC: Authorizing `identity:get_project(project_id=057afb8734e34b9687d7b36edb0034b4)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:13.268678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12809aa4-ce64-4cfd-b869-016143c6a363 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:13.270508 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 50/95] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:12 2022] GET /identity/v3/projects/057afb8734e34b9687d7b36edb0034b4 => generated 312 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:13.277295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12809aa4-ce64-4cfd-b869-016143c6a363 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:13.337386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e50bcb-cab9-403b-9234-dce4f62078df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:13.338008 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e50bcb-cab9-403b-9234-dce4f62078df None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:13.338297 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e50bcb-cab9-403b-9234-dce4f62078df None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:13.338583 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e50bcb-cab9-403b-9234-dce4f62078df None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:13.339459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e50bcb-cab9-403b-9234-dce4f62078df None admin] RBAC: Adding query filter params (role.id=48389faedba2470a885fae19084bb456, scope.project.id=5f976380e1494db480dc80868d51a0bc, user.id=18674c9f5ca2436b921d8f52a40444be) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:13.339994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e50bcb-cab9-403b-9234-dce4f62078df None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:13.342148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e50bcb-cab9-403b-9234-dce4f62078df None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:13.357430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 46/96] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:12 2022] GET /identity/v3/role_assignments?user.id=18674c9f5ca2436b921d8f52a40444be&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=48389faedba2470a885fae19084bb456 => generated 259 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:13.436987 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:13.484724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12809aa4-ce64-4cfd-b869-016143c6a363 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:13.547378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21d2d94-c705-45c5-b8c2-ce5402b9ab45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:13.548282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d2d94-c705-45c5-b8c2-ce5402b9ab45 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:13.548737 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d2d94-c705-45c5-b8c2-ce5402b9ab45 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:13.549157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d2d94-c705-45c5-b8c2-ce5402b9ab45 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:13.550056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21d2d94-c705-45c5-b8c2-ce5402b9ab45 None admin] RBAC: Adding query filter params (role.id=27476172dbb84c6e86b9070e02d06100, scope.project.id=057afb8734e34b9687d7b36edb0034b4, user.id=18674c9f5ca2436b921d8f52a40444be) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:13.550509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21d2d94-c705-45c5-b8c2-ce5402b9ab45 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:13.552160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21d2d94-c705-45c5-b8c2-ce5402b9ab45 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:13.563947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 51/97] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:13 2022] GET /identity/v3/role_assignments?user.id=18674c9f5ca2436b921d8f52a40444be&scope.project.id=057afb8734e34b9687d7b36edb0034b4&role.id=27476172dbb84c6e86b9070e02d06100 => generated 259 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:15.100730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea61a43-b87d-4bdf-8b10-b42ddd8b37e5 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:15.101258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea61a43-b87d-4bdf-8b10-b42ddd8b37e5 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:15.101332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea61a43-b87d-4bdf-8b10-b42ddd8b37e5 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:15.103232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 47/98] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:15.108735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e9c8d0-7ad4-4b2d-8c59-7c42b88c099c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:15.109157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e9c8d0-7ad4-4b2d-8c59-7c42b88c099c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:15.109279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e9c8d0-7ad4-4b2d-8c59-7c42b88c099c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:15.116908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a83359-1c80-4226-b18e-a792eaefb588 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:15.117438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a83359-1c80-4226-b18e-a792eaefb588 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:15.117946 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a83359-1c80-4226-b18e-a792eaefb588 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:15.120786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 48/99] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:22:15.126386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992599fd-f647-46d6-9f00-b5f8a7525cbc None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:15.126926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992599fd-f647-46d6-9f00-b5f8a7525cbc None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:15.127405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992599fd-f647-46d6-9f00-b5f8a7525cbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:15.133714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-62e9c8d0-7ad4-4b2d-8c59-7c42b88c099c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:15.136015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-62e9c8d0-7ad4-4b2d-8c59-7c42b88c099c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:15.159023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-992599fd-f647-46d6-9f00-b5f8a7525cbc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:15.161173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-992599fd-f647-46d6-9f00-b5f8a7525cbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:15.218576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 52/100] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:15.295255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 49/101] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:15.513984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e9c8d0-7ad4-4b2d-8c59-7c42b88c099c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:15.600921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:15.637616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e9c8d0-7ad4-4b2d-8c59-7c42b88c099c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:15.690660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a8066d-c592-4d64-881e-37308c275afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:15.691320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a8066d-c592-4d64-881e-37308c275afc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:15.691607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a8066d-c592-4d64-881e-37308c275afc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:15.691887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a8066d-c592-4d64-881e-37308c275afc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:15.693087 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a8066d-c592-4d64-881e-37308c275afc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:15.694614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a8066d-c592-4d64-881e-37308c275afc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:15.696258 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 53/102] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:15.702574 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992599fd-f647-46d6-9f00-b5f8a7525cbc None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:15.717781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ebaa5b-d8fe-4a6d-8d66-02f61997706a None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:15.718118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ebaa5b-d8fe-4a6d-8d66-02f61997706a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:15.718345 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ebaa5b-d8fe-4a6d-8d66-02f61997706a None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:15.719702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 54/103] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:15.726052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7161b2e4-af38-4c3f-85a3-a02ed09a87c4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:15.726334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7161b2e4-af38-4c3f-85a3-a02ed09a87c4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:15.726615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7161b2e4-af38-4c3f-85a3-a02ed09a87c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:15.756149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:15.762329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7161b2e4-af38-4c3f-85a3-a02ed09a87c4 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:15.763563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7161b2e4-af38-4c3f-85a3-a02ed09a87c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:15.841428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992599fd-f647-46d6-9f00-b5f8a7525cbc None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:15.870202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 55/104] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:15.871466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7161b2e4-af38-4c3f-85a3-a02ed09a87c4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:15.905728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa522db-f1c3-4403-b2b5-edb67162931f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:15.906350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa522db-f1c3-4403-b2b5-edb67162931f None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:15.906633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa522db-f1c3-4403-b2b5-edb67162931f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:15.906959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa522db-f1c3-4403-b2b5-edb67162931f None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:15.907664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa522db-f1c3-4403-b2b5-edb67162931f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:15.909299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa522db-f1c3-4403-b2b5-edb67162931f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:15.946907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:15.952367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 50/105] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:22:15 2022] POST /identity/v3/projects => generated 302 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:15.952367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff1bed-105f-4689-a176-90f320ef286f None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:15.952367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff1bed-105f-4689-a176-90f320ef286f None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:15.952367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eff1bed-105f-4689-a176-90f320ef286f None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:15.960581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 51/106] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:15.966975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a159084e-4073-4d24-abde-eabc317a55a5 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:15.967263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a159084e-4073-4d24-abde-eabc317a55a5 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:15.967542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a159084e-4073-4d24-abde-eabc317a55a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:16.021583 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-a159084e-4073-4d24-abde-eabc317a55a5 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:16.022840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a159084e-4073-4d24-abde-eabc317a55a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:16.031505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7161b2e4-af38-4c3f-85a3-a02ed09a87c4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:16.114224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3981bacf-7bf2-47f6-a299-381205ed7b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:16.115199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3981bacf-7bf2-47f6-a299-381205ed7b12 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:16.115199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3981bacf-7bf2-47f6-a299-381205ed7b12 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:16.115440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3981bacf-7bf2-47f6-a299-381205ed7b12 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:16.146263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3981bacf-7bf2-47f6-a299-381205ed7b12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-3981bacf-7bf2-47f6-a299-381205ed7b12 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:16.152211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:16.153092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:16.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:16.161697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 56/107] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:22:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:16.162317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 52/108] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:16.178037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3981bacf-7bf2-47f6-a299-381205ed7b12 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:16.268466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a159084e-4073-4d24-abde-eabc317a55a5 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:16.272688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:16.310670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3981bacf-7bf2-47f6-a299-381205ed7b12 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:16.357460 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:16.375231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc4ecec-f80d-44a2-8488-4b1fe627a09f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:16.375762 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc4ecec-f80d-44a2-8488-4b1fe627a09f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:16.376045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc4ecec-f80d-44a2-8488-4b1fe627a09f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:16.376321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc4ecec-f80d-44a2-8488-4b1fe627a09f None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:16.377022 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc4ecec-f80d-44a2-8488-4b1fe627a09f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:16.377331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc4ecec-f80d-44a2-8488-4b1fe627a09f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:16.378732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc4ecec-f80d-44a2-8488-4b1fe627a09f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:16.401293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 57/109] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:16.402745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc4ecec-f80d-44a2-8488-4b1fe627a09f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:16.463056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a159084e-4073-4d24-abde-eabc317a55a5 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:16.520835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be653085-be0b-4d63-8c09-8c0955cd9e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:16.521508 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be653085-be0b-4d63-8c09-8c0955cd9e2c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:16.521793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be653085-be0b-4d63-8c09-8c0955cd9e2c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:16.522066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be653085-be0b-4d63-8c09-8c0955cd9e2c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:16.561046 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be653085-be0b-4d63-8c09-8c0955cd9e2c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-be653085-be0b-4d63-8c09-8c0955cd9e2c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:16.566007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:16.566885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:16.567666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 53/110] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:22:16.569310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc4ecec-f80d-44a2-8488-4b1fe627a09f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:16.575376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be653085-be0b-4d63-8c09-8c0955cd9e2c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:16.619412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebd5917-2935-4c4c-a950-54090cec179d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:16.620527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebd5917-2935-4c4c-a950-54090cec179d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:16.621019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebd5917-2935-4c4c-a950-54090cec179d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:16.622249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebd5917-2935-4c4c-a950-54090cec179d None admin] PATH_INFO: `/v3/users/d47510704d9d4183ab8e8ff6578a4a83` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:16.624094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebd5917-2935-4c4c-a950-54090cec179d None admin] RBAC: Authorizing `identity:get_user(user_id=d47510704d9d4183ab8e8ff6578a4a83)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:16.627110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebd5917-2935-4c4c-a950-54090cec179d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:16.630090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 58/111] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:16 2022] GET /identity/v3/users/d47510704d9d4183ab8e8ff6578a4a83 => generated 247 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:16.631336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebd5917-2935-4c4c-a950-54090cec179d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:16.673767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:16.704767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:16.716938 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be653085-be0b-4d63-8c09-8c0955cd9e2c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:16.753442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b26b32c-56d4-47e6-86d8-a6ae11c8a7f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:16.754087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b26b32c-56d4-47e6-86d8-a6ae11c8a7f3 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:16.754369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b26b32c-56d4-47e6-86d8-a6ae11c8a7f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:16.754649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b26b32c-56d4-47e6-86d8-a6ae11c8a7f3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:16.757470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebd5917-2935-4c4c-a950-54090cec179d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:16.767767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b26b32c-56d4-47e6-86d8-a6ae11c8a7f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-9b26b32c-56d4-47e6-86d8-a6ae11c8a7f3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:16.773253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:16.774182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:16.775763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 54/112] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:16 2022] GET /identity/v3/roles/reader => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:16.781175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b26b32c-56d4-47e6-86d8-a6ae11c8a7f3 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:16.809448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77451a35-58e2-4d8b-8972-8a5a7b4b330f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:16.810578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77451a35-58e2-4d8b-8972-8a5a7b4b330f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:16.811226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77451a35-58e2-4d8b-8972-8a5a7b4b330f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:16.811704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77451a35-58e2-4d8b-8972-8a5a7b4b330f None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:16.812612 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77451a35-58e2-4d8b-8972-8a5a7b4b330f None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:16.813123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77451a35-58e2-4d8b-8972-8a5a7b4b330f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:16.814982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77451a35-58e2-4d8b-8972-8a5a7b4b330f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:16.825482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 59/113] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:16.826738 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77451a35-58e2-4d8b-8972-8a5a7b4b330f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:16.907092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b26b32c-56d4-47e6-86d8-a6ae11c8a7f3 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:16.932725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:16.937311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77451a35-58e2-4d8b-8972-8a5a7b4b330f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:16.944942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5db668-4a72-4336-a4f4-5d3c8ad2db80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:16.945576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5db668-4a72-4336-a4f4-5d3c8ad2db80 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:16.945862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5db668-4a72-4336-a4f4-5d3c8ad2db80 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:16.946142 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5db668-4a72-4336-a4f4-5d3c8ad2db80 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:16.948094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5db668-4a72-4336-a4f4-5d3c8ad2db80 None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:16.949601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5db668-4a72-4336-a4f4-5d3c8ad2db80 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:16.951407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 55/114] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:16 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:16.957231 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198ebaac-25d0-47b3-97c2-d25a472defaa None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:16.957517 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198ebaac-25d0-47b3-97c2-d25a472defaa None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:16.957797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198ebaac-25d0-47b3-97c2-d25a472defaa None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:16.958286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:16.959408 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 56/115] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:16 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:16.960427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198ebaac-25d0-47b3-97c2-d25a472defaa None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:16.979399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7593e83-056f-41b7-98a2-b4350534db68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:16.980415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7593e83-056f-41b7-98a2-b4350534db68 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:16.980707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7593e83-056f-41b7-98a2-b4350534db68 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:16.981144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7593e83-056f-41b7-98a2-b4350534db68 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:16.981822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7593e83-056f-41b7-98a2-b4350534db68 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:16.982237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7593e83-056f-41b7-98a2-b4350534db68 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:16.983537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7593e83-056f-41b7-98a2-b4350534db68 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:16.992104 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 60/116] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:16.998587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c91cfc-c0b8-4b5b-81b9-d668b1f8c74b None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:16.998876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c91cfc-c0b8-4b5b-81b9-d668b1f8c74b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:16.999164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c91cfc-c0b8-4b5b-81b9-d668b1f8c74b None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:17.000491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 61/117] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:17.001187 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c91cfc-c0b8-4b5b-81b9-d668b1f8c74b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:17.079589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198ebaac-25d0-47b3-97c2-d25a472defaa None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:17.113143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c91cfc-c0b8-4b5b-81b9-d668b1f8c74b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:17.116366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90134bcd-9764-4160-8056-a0705a52729b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:17.117016 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90134bcd-9764-4160-8056-a0705a52729b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:17.117393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90134bcd-9764-4160-8056-a0705a52729b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:17.117681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90134bcd-9764-4160-8056-a0705a52729b None admin] PATH_INFO: `/v3/users/18674c9f5ca2436b921d8f52a40444be` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:17.118845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90134bcd-9764-4160-8056-a0705a52729b None admin] RBAC: Authorizing `identity:get_user(user_id=18674c9f5ca2436b921d8f52a40444be)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:17.120308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90134bcd-9764-4160-8056-a0705a52729b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:17.126504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 57/118] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:16 2022] GET /identity/v3/users/18674c9f5ca2436b921d8f52a40444be => generated 275 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:17.126504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf83cff-e765-4d1d-b817-e571b2418b0d None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:17.126504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf83cff-e765-4d1d-b817-e571b2418b0d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:17.126504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf83cff-e765-4d1d-b817-e571b2418b0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:17.143425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-bdf83cff-e765-4d1d-b817-e571b2418b0d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:17.144675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-bdf83cff-e765-4d1d-b817-e571b2418b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:17.148913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ce2b59-493b-4885-b9aa-e633b246a9a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:17.150115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ce2b59-493b-4885-b9aa-e633b246a9a9 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:17.150405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ce2b59-493b-4885-b9aa-e633b246a9a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:17.150862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ce2b59-493b-4885-b9aa-e633b246a9a9 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc/users/d47510704d9d4183ab8e8ff6578a4a83/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:17.152823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ce2b59-493b-4885-b9aa-e633b246a9a9 None admin] RBAC: Authorizing `identity:create_grant(project_id=5f976380e1494db480dc80868d51a0bc, user_id=d47510704d9d4183ab8e8ff6578a4a83, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:17.154288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ce2b59-493b-4885-b9aa-e633b246a9a9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:17.167661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 62/119] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:22:17 2022] PUT /identity/v3/projects/5f976380e1494db480dc80868d51a0bc/users/d47510704d9d4183ab8e8ff6578a4a83/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:17.168477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ce2b59-493b-4885-b9aa-e633b246a9a9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:17.224936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:17.241100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:17.262475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 58/120] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:17.264088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f866cef4-7f00-43ea-b9bf-ee8b3945b2eb None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:17.264552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f866cef4-7f00-43ea-b9bf-ee8b3945b2eb None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:17.264955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f866cef4-7f00-43ea-b9bf-ee8b3945b2eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:17.287146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-f866cef4-7f00-43ea-b9bf-ee8b3945b2eb None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:17.288790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f866cef4-7f00-43ea-b9bf-ee8b3945b2eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:17.342937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ce2b59-493b-4885-b9aa-e633b246a9a9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:17.380053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 59/121] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:17.381491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f866cef4-7f00-43ea-b9bf-ee8b3945b2eb None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:17.392091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11e9141-606f-41a1-a314-b51973ffd45b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:17.392971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11e9141-606f-41a1-a314-b51973ffd45b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:17.393400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11e9141-606f-41a1-a314-b51973ffd45b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:17.393769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11e9141-606f-41a1-a314-b51973ffd45b None admin] PATH_INFO: `/v3/users/07c216b73cfd4842a37c9f2bb1e2051d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:17.411284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11e9141-606f-41a1-a314-b51973ffd45b None admin] RBAC: Authorizing `identity:get_user(user_id=07c216b73cfd4842a37c9f2bb1e2051d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:17.413171 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11e9141-606f-41a1-a314-b51973ffd45b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:17.416783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 63/122] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:17 2022] GET /identity/v3/users/07c216b73cfd4842a37c9f2bb1e2051d => generated 289 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:17.424426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11e9141-606f-41a1-a314-b51973ffd45b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:17.556588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f866cef4-7f00-43ea-b9bf-ee8b3945b2eb None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:17.581861 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:17.606145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:17.613710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11e9141-606f-41a1-a314-b51973ffd45b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:17.617065 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca682cb-393c-4589-a18e-4759494f6c04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:17.617733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca682cb-393c-4589-a18e-4759494f6c04 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:17.618021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca682cb-393c-4589-a18e-4759494f6c04 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:17.618959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca682cb-393c-4589-a18e-4759494f6c04 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:17.620298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca682cb-393c-4589-a18e-4759494f6c04 None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:17.621792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca682cb-393c-4589-a18e-4759494f6c04 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:17.623639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 60/123] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:17 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:17.629052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca682cb-393c-4589-a18e-4759494f6c04 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:17.708300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f2fb63-2a57-43b5-8a14-fe289ae8e894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:17.708956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f2fb63-2a57-43b5-8a14-fe289ae8e894 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:17.709244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f2fb63-2a57-43b5-8a14-fe289ae8e894 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:17.709528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f2fb63-2a57-43b5-8a14-fe289ae8e894 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:17.769979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca682cb-393c-4589-a18e-4759494f6c04 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:17.836750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cd724e-a59c-42ba-a9c9-c88085a0fc19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:17.838719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cd724e-a59c-42ba-a9c9-c88085a0fc19 None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:17.839757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cd724e-a59c-42ba-a9c9-c88085a0fc19 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:17.840242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cd724e-a59c-42ba-a9c9-c88085a0fc19 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc/users/18674c9f5ca2436b921d8f52a40444be/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:17.841459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f2fb63-2a57-43b5-8a14-fe289ae8e894 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:17.842657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4cd724e-a59c-42ba-a9c9-c88085a0fc19 None admin] RBAC: Authorizing `identity:create_grant(project_id=5f976380e1494db480dc80868d51a0bc, user_id=18674c9f5ca2436b921d8f52a40444be, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:17.842968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f2fb63-2a57-43b5-8a14-fe289ae8e894 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:17.844672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4cd724e-a59c-42ba-a9c9-c88085a0fc19 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:17.868025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 61/124] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:22:17 2022] PUT /identity/v3/projects/5f976380e1494db480dc80868d51a0bc/users/18674c9f5ca2436b921d8f52a40444be/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:22:17.869499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cd724e-a59c-42ba-a9c9-c88085a0fc19 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:17.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:18.018742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cd724e-a59c-42ba-a9c9-c88085a0fc19 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:18.038073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:18.047077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 64/125] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 625 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 26 01:22:18.051283 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f2fb63-2a57-43b5-8a14-fe289ae8e894 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:18.114584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab60eaa9-5e0d-4b2f-a162-667dce0d6b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:18.115250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab60eaa9-5e0d-4b2f-a162-667dce0d6b8e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:18.115535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab60eaa9-5e0d-4b2f-a162-667dce0d6b8e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:18.115813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab60eaa9-5e0d-4b2f-a162-667dce0d6b8e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:18.145370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab60eaa9-5e0d-4b2f-a162-667dce0d6b8e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ab60eaa9-5e0d-4b2f-a162-667dce0d6b8e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:18.151620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:18.152808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:18.153588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:18.159244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 62/126] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:18.160004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab60eaa9-5e0d-4b2f-a162-667dce0d6b8e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:18.186638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f2fb63-2a57-43b5-8a14-fe289ae8e894 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:18.236596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20a23b6-3320-4769-a136-49d198aa108c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:18.237599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20a23b6-3320-4769-a136-49d198aa108c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:18.238326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20a23b6-3320-4769-a136-49d198aa108c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:18.238796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20a23b6-3320-4769-a136-49d198aa108c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:18.254027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20a23b6-3320-4769-a136-49d198aa108c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b20a23b6-3320-4769-a136-49d198aa108c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:18.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:18.262947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:18.263715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:18.265883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 65/127] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:18.269180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20a23b6-3320-4769-a136-49d198aa108c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:18.310374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab60eaa9-5e0d-4b2f-a162-667dce0d6b8e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:18.333502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:18.345276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:18.395413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20a23b6-3320-4769-a136-49d198aa108c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:18.433175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4087228-432a-46d6-b6fc-de993dc2ad90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:18.433827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4087228-432a-46d6-b6fc-de993dc2ad90 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:18.434115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4087228-432a-46d6-b6fc-de993dc2ad90 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:18.434397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4087228-432a-46d6-b6fc-de993dc2ad90 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:18.435906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e41834a-4a26-40b6-b0ff-9d758cbf232a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:18.436742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e41834a-4a26-40b6-b0ff-9d758cbf232a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:18.437237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e41834a-4a26-40b6-b0ff-9d758cbf232a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:18.437700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e41834a-4a26-40b6-b0ff-9d758cbf232a None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:18.438554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e41834a-4a26-40b6-b0ff-9d758cbf232a None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:18.439050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e41834a-4a26-40b6-b0ff-9d758cbf232a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:18.440724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e41834a-4a26-40b6-b0ff-9d758cbf232a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:18.441586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4087228-432a-46d6-b6fc-de993dc2ad90 None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:18.444110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4087228-432a-46d6-b6fc-de993dc2ad90 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:18.451038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 63/128] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:18 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:18.452094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4087228-432a-46d6-b6fc-de993dc2ad90 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:18.459003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 66/129] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:18.464666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e41834a-4a26-40b6-b0ff-9d758cbf232a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:18.577897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4087228-432a-46d6-b6fc-de993dc2ad90 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:18.627336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf1ca65-85b9-46ab-8b1b-0d23b01cae60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:18.628380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1ca65-85b9-46ab-8b1b-0d23b01cae60 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:18.628869 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1ca65-85b9-46ab-8b1b-0d23b01cae60 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:18.629366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1ca65-85b9-46ab-8b1b-0d23b01cae60 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:18.630241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf1ca65-85b9-46ab-8b1b-0d23b01cae60 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:18.630749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf1ca65-85b9-46ab-8b1b-0d23b01cae60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:18.632317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf1ca65-85b9-46ab-8b1b-0d23b01cae60 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:18.646007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 64/130] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:18.647341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf1ca65-85b9-46ab-8b1b-0d23b01cae60 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:18.697219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e41834a-4a26-40b6-b0ff-9d758cbf232a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:18.704619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:18.745342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:18.779278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf1ca65-85b9-46ab-8b1b-0d23b01cae60 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:18.805086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2e07f0-b9be-49e1-b9ed-7f6f05355725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:18.805741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e07f0-b9be-49e1-b9ed-7f6f05355725 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:18.806335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e07f0-b9be-49e1-b9ed-7f6f05355725 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:18.806335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e07f0-b9be-49e1-b9ed-7f6f05355725 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:18.807080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2e07f0-b9be-49e1-b9ed-7f6f05355725 None admin] RBAC: Adding query filter params (role.id=8922a4a1fc6e4b22815abb10e1f19807, scope.project.id=5f976380e1494db480dc80868d51a0bc, user.id=07c216b73cfd4842a37c9f2bb1e2051d) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:18.807394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2e07f0-b9be-49e1-b9ed-7f6f05355725 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:18.808879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2e07f0-b9be-49e1-b9ed-7f6f05355725 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:18.814803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 67/131] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:18 2022] GET /identity/v3/role_assignments?user.id=07c216b73cfd4842a37c9f2bb1e2051d&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=8922a4a1fc6e4b22815abb10e1f19807 => generated 259 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:18.815551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2e07f0-b9be-49e1-b9ed-7f6f05355725 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:18.819832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c50406f-ae62-4fe3-bca7-d8c138d63eb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:18.821018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50406f-ae62-4fe3-bca7-d8c138d63eb6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:18.822103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50406f-ae62-4fe3-bca7-d8c138d63eb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:18.822626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50406f-ae62-4fe3-bca7-d8c138d63eb6 None admin] PATH_INFO: `/v3/users/18674c9f5ca2436b921d8f52a40444be` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:18.824271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c50406f-ae62-4fe3-bca7-d8c138d63eb6 None admin] RBAC: Authorizing `identity:get_user(user_id=18674c9f5ca2436b921d8f52a40444be)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:18.826216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c50406f-ae62-4fe3-bca7-d8c138d63eb6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:18.828400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 65/132] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:18 2022] GET /identity/v3/users/18674c9f5ca2436b921d8f52a40444be => generated 275 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:18.830223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660601b6-f7eb-46d9-b279-0e42563270dc None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:18.830712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660601b6-f7eb-46d9-b279-0e42563270dc None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:18.831188 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660601b6-f7eb-46d9-b279-0e42563270dc None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:18.833042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 66/133] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:18.834796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660601b6-f7eb-46d9-b279-0e42563270dc None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:18.927072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2e07f0-b9be-49e1-b9ed-7f6f05355725 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:18.980921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660601b6-f7eb-46d9-b279-0e42563270dc None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:18.993048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13be9ba6-73a6-41ef-833c-5f6aec2a75b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:18.994537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be9ba6-73a6-41ef-833c-5f6aec2a75b7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:18.994985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be9ba6-73a6-41ef-833c-5f6aec2a75b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:18.995415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be9ba6-73a6-41ef-833c-5f6aec2a75b7 None admin] PATH_INFO: `/v3/users/18674c9f5ca2436b921d8f52a40444be` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:18.996710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:18.996847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13be9ba6-73a6-41ef-833c-5f6aec2a75b7 None admin] RBAC: Authorizing `identity:get_user(user_id=18674c9f5ca2436b921d8f52a40444be)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:18.998320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13be9ba6-73a6-41ef-833c-5f6aec2a75b7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:19.000160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 68/134] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:18 2022] GET /identity/v3/users/18674c9f5ca2436b921d8f52a40444be => generated 275 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:19.001305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a123ebc4-89f6-4036-9cda-db5266c53d7a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:19.004128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a123ebc4-89f6-4036-9cda-db5266c53d7a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:19.004596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a123ebc4-89f6-4036-9cda-db5266c53d7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:19.043616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a123ebc4-89f6-4036-9cda-db5266c53d7a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:19.046809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a123ebc4-89f6-4036-9cda-db5266c53d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:19.053449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4651b64c-7d9d-4970-923b-4187daf1df45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:19.054116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4651b64c-7d9d-4970-923b-4187daf1df45 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:19.054402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4651b64c-7d9d-4970-923b-4187daf1df45 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:19.054682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4651b64c-7d9d-4970-923b-4187daf1df45 None admin] PATH_INFO: `/v3/projects/057afb8734e34b9687d7b36edb0034b4` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:19.058542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4651b64c-7d9d-4970-923b-4187daf1df45 None admin] RBAC: Authorizing `identity:get_project(project_id=057afb8734e34b9687d7b36edb0034b4)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:19.060045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4651b64c-7d9d-4970-923b-4187daf1df45 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:19.061816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 67/135] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:18 2022] GET /identity/v3/projects/057afb8734e34b9687d7b36edb0034b4 => generated 312 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:19.062611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4651b64c-7d9d-4970-923b-4187daf1df45 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:19.126722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:19.159003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 69/136] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:19.159851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a123ebc4-89f6-4036-9cda-db5266c53d7a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:19.244950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4651b64c-7d9d-4970-923b-4187daf1df45 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:19.316303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740dcdfb-d64a-46a9-a76a-9e3e6d01a74c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:19.317086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740dcdfb-d64a-46a9-a76a-9e3e6d01a74c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:19.317287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740dcdfb-d64a-46a9-a76a-9e3e6d01a74c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:19.317568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740dcdfb-d64a-46a9-a76a-9e3e6d01a74c None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:19.319851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740dcdfb-d64a-46a9-a76a-9e3e6d01a74c None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:19.321390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740dcdfb-d64a-46a9-a76a-9e3e6d01a74c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:19.324455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 68/137] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:19 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:19.328882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740dcdfb-d64a-46a9-a76a-9e3e6d01a74c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:19.365524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a123ebc4-89f6-4036-9cda-db5266c53d7a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:19.420814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:19.426897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a278665-33f4-4291-8265-b0254b0b7ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:19.427680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a278665-33f4-4291-8265-b0254b0b7ee8 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:19.427898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a278665-33f4-4291-8265-b0254b0b7ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:19.428173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a278665-33f4-4291-8265-b0254b0b7ee8 None admin] PATH_INFO: `/v3/projects/057afb8734e34b9687d7b36edb0034b4/users/18674c9f5ca2436b921d8f52a40444be/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:19.430158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a278665-33f4-4291-8265-b0254b0b7ee8 None admin] RBAC: Authorizing `identity:create_grant(project_id=057afb8734e34b9687d7b36edb0034b4, user_id=18674c9f5ca2436b921d8f52a40444be, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:19.432197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a278665-33f4-4291-8265-b0254b0b7ee8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:19.465250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 70/138] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:22:19 2022] PUT /identity/v3/projects/057afb8734e34b9687d7b36edb0034b4/users/18674c9f5ca2436b921d8f52a40444be/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:19.501739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a278665-33f4-4291-8265-b0254b0b7ee8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:19.542641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740dcdfb-d64a-46a9-a76a-9e3e6d01a74c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:19.597541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e27921-56ed-444c-bdf9-900269eff3a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:19.598219 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e27921-56ed-444c-bdf9-900269eff3a4 None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:19.598500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e27921-56ed-444c-bdf9-900269eff3a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:19.598776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e27921-56ed-444c-bdf9-900269eff3a4 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc/users/18674c9f5ca2436b921d8f52a40444be/roles/48389faedba2470a885fae19084bb456` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:19.600995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e27921-56ed-444c-bdf9-900269eff3a4 None admin] RBAC: Authorizing `identity:create_grant(project_id=5f976380e1494db480dc80868d51a0bc, user_id=18674c9f5ca2436b921d8f52a40444be, role_id=48389faedba2470a885fae19084bb456)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:19.602532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e27921-56ed-444c-bdf9-900269eff3a4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:19.610528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:19.626908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 69/139] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:22:19 2022] PUT /identity/v3/projects/5f976380e1494db480dc80868d51a0bc/users/18674c9f5ca2436b921d8f52a40444be/roles/48389faedba2470a885fae19084bb456 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:19.690547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a278665-33f4-4291-8265-b0254b0b7ee8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:19.731384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0d1002-43a6-489c-965b-49fcacdf1d44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:19.732070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d1002-43a6-489c-965b-49fcacdf1d44 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:19.732369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d1002-43a6-489c-965b-49fcacdf1d44 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:19.732655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d1002-43a6-489c-965b-49fcacdf1d44 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:19.733909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0d1002-43a6-489c-965b-49fcacdf1d44 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:19.735462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0d1002-43a6-489c-965b-49fcacdf1d44 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:19.737206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 71/140] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:19.742779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e27921-56ed-444c-bdf9-900269eff3a4 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:19.862413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e27921-56ed-444c-bdf9-900269eff3a4 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:19.864071 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:19.901214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c108ab-d285-4dde-ad44-3786c12fbcbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:19.901901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c108ab-d285-4dde-ad44-3786c12fbcbd None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:19.902186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c108ab-d285-4dde-ad44-3786c12fbcbd None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:19.902512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c108ab-d285-4dde-ad44-3786c12fbcbd None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:19.903310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c108ab-d285-4dde-ad44-3786c12fbcbd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:19.905239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c108ab-d285-4dde-ad44-3786c12fbcbd None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:19.938191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e181a2-8189-4d12-9370-eec143f428b7 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:19.938513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e181a2-8189-4d12-9370-eec143f428b7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:19.938743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e181a2-8189-4d12-9370-eec143f428b7 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:19.940145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 72/141] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:22:19.943036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 70/142] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:22:19 2022] POST /identity/v3/users => generated 283 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:19.945818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1028d74c-61dd-498b-8a4a-6cda01e4a170 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:19.946153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1028d74c-61dd-498b-8a4a-6cda01e4a170 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:19.946507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1028d74c-61dd-498b-8a4a-6cda01e4a170 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:19.970126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1028d74c-61dd-498b-8a4a-6cda01e4a170 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:19.971355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1028d74c-61dd-498b-8a4a-6cda01e4a170 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:20.056377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 73/143] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:20.405912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c108ab-d285-4dde-ad44-3786c12fbcbd None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:20.512750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c108ab-d285-4dde-ad44-3786c12fbcbd None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:20.547291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db83e91-9d45-463b-b98e-db8e55f8c7c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:20.547932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db83e91-9d45-463b-b98e-db8e55f8c7c0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:20.548216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db83e91-9d45-463b-b98e-db8e55f8c7c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:20.548522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db83e91-9d45-463b-b98e-db8e55f8c7c0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:20.559005 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db83e91-9d45-463b-b98e-db8e55f8c7c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-6db83e91-9d45-463b-b98e-db8e55f8c7c0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:20.564312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:20.565241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:20.566058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 71/144] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:22:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:20.572158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1028d74c-61dd-498b-8a4a-6cda01e4a170 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:20.615820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:20.678832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1028d74c-61dd-498b-8a4a-6cda01e4a170 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:20.718558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cf2979-ea52-4d21-88bc-23de4a20fa81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:20.719168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cf2979-ea52-4d21-88bc-23de4a20fa81 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:20.719446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cf2979-ea52-4d21-88bc-23de4a20fa81 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:20.719717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cf2979-ea52-4d21-88bc-23de4a20fa81 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:20.720406 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cf2979-ea52-4d21-88bc-23de4a20fa81 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:20.720713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cf2979-ea52-4d21-88bc-23de4a20fa81 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:20.722101 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cf2979-ea52-4d21-88bc-23de4a20fa81 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:20.731651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 74/145] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:20.737189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db83e91-9d45-463b-b98e-db8e55f8c7c0 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:20.760809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:20.822403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d814fe47-e0ec-4e78-9c97-8af9b64ce6bd None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:20.822615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d814fe47-e0ec-4e78-9c97-8af9b64ce6bd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:20.822855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d814fe47-e0ec-4e78-9c97-8af9b64ce6bd None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:20.824192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 75/146] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:22:20.829822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086c861b-bfba-43bc-b2dd-891fe666d07f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:20.830100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086c861b-bfba-43bc-b2dd-891fe666d07f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:20.830368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086c861b-bfba-43bc-b2dd-891fe666d07f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:20.866925 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-086c861b-bfba-43bc-b2dd-891fe666d07f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:20.868118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-086c861b-bfba-43bc-b2dd-891fe666d07f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:20.893680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db83e91-9d45-463b-b98e-db8e55f8c7c0 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:20.973871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d7df60-b5f5-4d82-af20-867ae557ab0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:20.974555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d7df60-b5f5-4d82-af20-867ae557ab0a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:20.974843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d7df60-b5f5-4d82-af20-867ae557ab0a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:20.975118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d7df60-b5f5-4d82-af20-867ae557ab0a None admin] PATH_INFO: `/v3/users/d47510704d9d4183ab8e8ff6578a4a83` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:20.977094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d7df60-b5f5-4d82-af20-867ae557ab0a None admin] RBAC: Authorizing `identity:get_user(user_id=d47510704d9d4183ab8e8ff6578a4a83)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:20.979599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d7df60-b5f5-4d82-af20-867ae557ab0a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:20.981517 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 72/147] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:20 2022] GET /identity/v3/users/d47510704d9d4183ab8e8ff6578a4a83 => generated 247 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:20.988942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d7df60-b5f5-4d82-af20-867ae557ab0a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:21.018755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 76/148] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:21.096850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:21.112215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d7df60-b5f5-4d82-af20-867ae557ab0a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:21.154896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4bbab9-33d6-4c2c-8c94-9681fdc53ea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:21.155509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4bbab9-33d6-4c2c-8c94-9681fdc53ea8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:21.155789 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4bbab9-33d6-4c2c-8c94-9681fdc53ea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:21.156063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4bbab9-33d6-4c2c-8c94-9681fdc53ea8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:21.246018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4bbab9-33d6-4c2c-8c94-9681fdc53ea8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:21.247433 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4bbab9-33d6-4c2c-8c94-9681fdc53ea8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:21.349000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:21.368356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 73/149] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 26 01:22:21.374231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086c861b-bfba-43bc-b2dd-891fe666d07f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:21.386664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4bbab9-33d6-4c2c-8c94-9681fdc53ea8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:21.432721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:21.558270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4bbab9-33d6-4c2c-8c94-9681fdc53ea8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:21.570455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086c861b-bfba-43bc-b2dd-891fe666d07f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:21.601986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5da1ca-f2fb-4db3-be9d-b7d500751442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:21.602698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5da1ca-f2fb-4db3-be9d-b7d500751442 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:21.602975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5da1ca-f2fb-4db3-be9d-b7d500751442 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:21.603249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5da1ca-f2fb-4db3-be9d-b7d500751442 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:21.615500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5da1ca-f2fb-4db3-be9d-b7d500751442 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-0b5da1ca-f2fb-4db3-be9d-b7d500751442 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:21.623903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:21.624901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:21.626056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 74/150] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:21.631460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5da1ca-f2fb-4db3-be9d-b7d500751442 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:21.637578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccae6f93-cf23-4aa1-95c1-8fa44b43cb2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:21.638227 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccae6f93-cf23-4aa1-95c1-8fa44b43cb2e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:21.638508 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccae6f93-cf23-4aa1-95c1-8fa44b43cb2e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:21.638785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccae6f93-cf23-4aa1-95c1-8fa44b43cb2e None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:21.639938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccae6f93-cf23-4aa1-95c1-8fa44b43cb2e None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:21.641429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccae6f93-cf23-4aa1-95c1-8fa44b43cb2e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:21.643194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 77/151] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:21 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:21.669142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccae6f93-cf23-4aa1-95c1-8fa44b43cb2e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:21.689682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:21.755944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5da1ca-f2fb-4db3-be9d-b7d500751442 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:21.796431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1660954-cbd1-4238-b3ea-d4c2288a5c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:21.797282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1660954-cbd1-4238-b3ea-d4c2288a5c3d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:21.797624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1660954-cbd1-4238-b3ea-d4c2288a5c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:21.797959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1660954-cbd1-4238-b3ea-d4c2288a5c3d None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:21.798698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1660954-cbd1-4238-b3ea-d4c2288a5c3d None admin] RBAC: Adding query filter params (name=member) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:21.799070 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1660954-cbd1-4238-b3ea-d4c2288a5c3d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:21.800880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1660954-cbd1-4238-b3ea-d4c2288a5c3d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:21.811035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 75/152] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:21.817707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1660954-cbd1-4238-b3ea-d4c2288a5c3d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:21.872940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccae6f93-cf23-4aa1-95c1-8fa44b43cb2e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:21.876679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:21.922134 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b97c18b-3f99-4ccf-8099-cb3168afd533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:21.922755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97c18b-3f99-4ccf-8099-cb3168afd533 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:21.923033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97c18b-3f99-4ccf-8099-cb3168afd533 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:21.923301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97c18b-3f99-4ccf-8099-cb3168afd533 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:21.924035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b97c18b-3f99-4ccf-8099-cb3168afd533 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=5f976380e1494db480dc80868d51a0bc, user.id=d47510704d9d4183ab8e8ff6578a4a83) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:21.924369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b97c18b-3f99-4ccf-8099-cb3168afd533 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:21.925943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b97c18b-3f99-4ccf-8099-cb3168afd533 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:21.933046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 78/153] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:21 2022] GET /identity/v3/role_assignments?user.id=d47510704d9d4183ab8e8ff6578a4a83&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 608 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:21.934651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a126b3a-3c7f-4c53-9fda-6f836a8cd0fb None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:21.935046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a126b3a-3c7f-4c53-9fda-6f836a8cd0fb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:21.935328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a126b3a-3c7f-4c53-9fda-6f836a8cd0fb None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:21.938119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 79/154] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:21.940683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1660954-cbd1-4238-b3ea-d4c2288a5c3d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:21.943723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284887d8-810a-452c-b2ef-0127c5567dcb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:21.944014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284887d8-810a-452c-b2ef-0127c5567dcb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:21.944294 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284887d8-810a-452c-b2ef-0127c5567dcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:21.952262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:21.979078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-284887d8-810a-452c-b2ef-0127c5567dcb None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:21.980337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-284887d8-810a-452c-b2ef-0127c5567dcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:22.011472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74854026-80e7-4d04-be1f-c14e8f8baa25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:22.012128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74854026-80e7-4d04-be1f-c14e8f8baa25 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:22.012516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74854026-80e7-4d04-be1f-c14e8f8baa25 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:22.012794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74854026-80e7-4d04-be1f-c14e8f8baa25 None admin] PATH_INFO: `/v3/users/18674c9f5ca2436b921d8f52a40444be` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:22.013996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74854026-80e7-4d04-be1f-c14e8f8baa25 None admin] RBAC: Authorizing `identity:get_user(user_id=18674c9f5ca2436b921d8f52a40444be)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:22.015503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74854026-80e7-4d04-be1f-c14e8f8baa25 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:22.017518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 76/155] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:21 2022] GET /identity/v3/users/18674c9f5ca2436b921d8f52a40444be => generated 275 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:22.022964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74854026-80e7-4d04-be1f-c14e8f8baa25 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:22.139808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 80/156] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:22.190917 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74854026-80e7-4d04-be1f-c14e8f8baa25 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:22.227636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a7b2f5-8e95-4303-8c79-b2b17c4a59e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:22.228520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a7b2f5-8e95-4303-8c79-b2b17c4a59e6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:22.228936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a7b2f5-8e95-4303-8c79-b2b17c4a59e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:22.229214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a7b2f5-8e95-4303-8c79-b2b17c4a59e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:22.283496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:22.317174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a7b2f5-8e95-4303-8c79-b2b17c4a59e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:22.318661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a7b2f5-8e95-4303-8c79-b2b17c4a59e6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:22.437474 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 77/157] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 26 01:22:22.443015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284887d8-810a-452c-b2ef-0127c5567dcb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:22.547029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:22.554426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284887d8-810a-452c-b2ef-0127c5567dcb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:22.597060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca26b82-0bfc-4f30-b134-6318d2719de5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:22.597714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca26b82-0bfc-4f30-b134-6318d2719de5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:22.597996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca26b82-0bfc-4f30-b134-6318d2719de5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:22.598328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca26b82-0bfc-4f30-b134-6318d2719de5 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:22.599500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca26b82-0bfc-4f30-b134-6318d2719de5 None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:22.601258 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca26b82-0bfc-4f30-b134-6318d2719de5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:22.604557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 81/158] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:22 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:22.610417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a7b2f5-8e95-4303-8c79-b2b17c4a59e6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:22.703329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:22.732974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a7b2f5-8e95-4303-8c79-b2b17c4a59e6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:22.769606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de969ed4-6f2c-4406-bd17-e38259a02187 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:22.770215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de969ed4-6f2c-4406-bd17-e38259a02187 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:22.770498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de969ed4-6f2c-4406-bd17-e38259a02187 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:22.770747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de969ed4-6f2c-4406-bd17-e38259a02187 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:22.771482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de969ed4-6f2c-4406-bd17-e38259a02187 None admin] RBAC: Adding query filter params (role.id=27476172dbb84c6e86b9070e02d06100, scope.project.id=5f976380e1494db480dc80868d51a0bc, user.id=18674c9f5ca2436b921d8f52a40444be) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:22.771795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de969ed4-6f2c-4406-bd17-e38259a02187 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:22.773252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de969ed4-6f2c-4406-bd17-e38259a02187 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:22.782045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca26b82-0bfc-4f30-b134-6318d2719de5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:22.782820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 78/159] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:22 2022] GET /identity/v3/role_assignments?user.id=18674c9f5ca2436b921d8f52a40444be&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=27476172dbb84c6e86b9070e02d06100 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:22.919335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca26b82-0bfc-4f30-b134-6318d2719de5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:22.956860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43f3b2c-8a38-4989-bf7b-ff2ffc690426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:22.957394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43f3b2c-8a38-4989-bf7b-ff2ffc690426 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:22.957668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43f3b2c-8a38-4989-bf7b-ff2ffc690426 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:22.957935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43f3b2c-8a38-4989-bf7b-ff2ffc690426 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:22.970173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43f3b2c-8a38-4989-bf7b-ff2ffc690426 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f43f3b2c-8a38-4989-bf7b-ff2ffc690426 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:22.975225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:22.977079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:22.979178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 82/160] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:22.989906 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de969ed4-6f2c-4406-bd17-e38259a02187 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:23.162428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de969ed4-6f2c-4406-bd17-e38259a02187 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:23.206196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778e72e1-f755-43fc-9a14-196cc0598e0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:23.207131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778e72e1-f755-43fc-9a14-196cc0598e0c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:23.207612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778e72e1-f755-43fc-9a14-196cc0598e0c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:23.208033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778e72e1-f755-43fc-9a14-196cc0598e0c None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:23.208879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778e72e1-f755-43fc-9a14-196cc0598e0c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:23.209324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778e72e1-f755-43fc-9a14-196cc0598e0c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:23.210862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778e72e1-f755-43fc-9a14-196cc0598e0c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:23.224241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 79/161] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:23.232679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43f3b2c-8a38-4989-bf7b-ff2ffc690426 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:23.271468 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:23.375120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43f3b2c-8a38-4989-bf7b-ff2ffc690426 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:23.420942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa11e2-89da-4014-9c31-328f5ec6bb05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:23.421883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa11e2-89da-4014-9c31-328f5ec6bb05 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:23.422316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa11e2-89da-4014-9c31-328f5ec6bb05 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:23.422739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa11e2-89da-4014-9c31-328f5ec6bb05 None admin] PATH_INFO: `/v3/users/07c216b73cfd4842a37c9f2bb1e2051d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:23.429030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76aa11e2-89da-4014-9c31-328f5ec6bb05 None admin] RBAC: Authorizing `identity:get_user(user_id=07c216b73cfd4842a37c9f2bb1e2051d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:23.430677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76aa11e2-89da-4014-9c31-328f5ec6bb05 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:23.432920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 83/162] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:23 2022] GET /identity/v3/users/07c216b73cfd4842a37c9f2bb1e2051d => generated 289 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:23.441304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778e72e1-f755-43fc-9a14-196cc0598e0c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:23.444450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:23.563598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778e72e1-f755-43fc-9a14-196cc0598e0c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:23.601753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af01d844-d891-45e8-8e14-242f641464b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:23.603169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af01d844-d891-45e8-8e14-242f641464b4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:23.603605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af01d844-d891-45e8-8e14-242f641464b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:23.604018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af01d844-d891-45e8-8e14-242f641464b4 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:23.605391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af01d844-d891-45e8-8e14-242f641464b4 None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:23.606997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af01d844-d891-45e8-8e14-242f641464b4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:23.608802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 80/163] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:23 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:23.617221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa11e2-89da-4014-9c31-328f5ec6bb05 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:23.733990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:23.753124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa11e2-89da-4014-9c31-328f5ec6bb05 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:23.799348 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbda4df-2d18-4c21-9979-e7d5401add10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:23.800713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbda4df-2d18-4c21-9979-e7d5401add10 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:23.801157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbda4df-2d18-4c21-9979-e7d5401add10 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:23.801569 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbda4df-2d18-4c21-9979-e7d5401add10 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc/users/07c216b73cfd4842a37c9f2bb1e2051d/roles/8922a4a1fc6e4b22815abb10e1f19807` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:23.803928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbda4df-2d18-4c21-9979-e7d5401add10 None admin] RBAC: Authorizing `identity:create_grant(project_id=5f976380e1494db480dc80868d51a0bc, user_id=07c216b73cfd4842a37c9f2bb1e2051d, role_id=8922a4a1fc6e4b22815abb10e1f19807)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:23.805578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbda4df-2d18-4c21-9979-e7d5401add10 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:23.821005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 84/164] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:22:23 2022] PUT /identity/v3/projects/5f976380e1494db480dc80868d51a0bc/users/07c216b73cfd4842a37c9f2bb1e2051d/roles/8922a4a1fc6e4b22815abb10e1f19807 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:22:24.125640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf29124-2d10-4e87-b25e-e631828945d1 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:24.126563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf29124-2d10-4e87-b25e-e631828945d1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:24.127049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf29124-2d10-4e87-b25e-e631828945d1 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:24.128614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 81/165] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:24.133901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960b47de-be18-44cf-93cd-0b7de405856f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:24.134722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960b47de-be18-44cf-93cd-0b7de405856f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:24.135159 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960b47de-be18-44cf-93cd-0b7de405856f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:24.160630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-960b47de-be18-44cf-93cd-0b7de405856f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:24.161827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-960b47de-be18-44cf-93cd-0b7de405856f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:24.248529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 85/166] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:24.529066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45846977-c48d-4fc8-9fa9-9e686505886c None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:24.529747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45846977-c48d-4fc8-9fa9-9e686505886c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:24.530232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45846977-c48d-4fc8-9fa9-9e686505886c None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:24.531760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 82/167] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:24.537593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c891fc9-fbdc-4c02-81fa-eaa2ad7657f9 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:24.537911 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c891fc9-fbdc-4c02-81fa-eaa2ad7657f9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:24.538148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c891fc9-fbdc-4c02-81fa-eaa2ad7657f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:24.569390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4c891fc9-fbdc-4c02-81fa-eaa2ad7657f9 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:24.570592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4c891fc9-fbdc-4c02-81fa-eaa2ad7657f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:24.658023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 86/168] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:24.748552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45846977-c48d-4fc8-9fa9-9e686505886c None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:24.842172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:24.879109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45846977-c48d-4fc8-9fa9-9e686505886c None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:24.916756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d50728-3f77-40cf-9702-8c572d15cbe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:24.917767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d50728-3f77-40cf-9702-8c572d15cbe3 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:24.918294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d50728-3f77-40cf-9702-8c572d15cbe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:24.918784 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d50728-3f77-40cf-9702-8c572d15cbe3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:24.934607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d50728-3f77-40cf-9702-8c572d15cbe3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-25d50728-3f77-40cf-9702-8c572d15cbe3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:24.939948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:24.945030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:24.945888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 83/169] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:24.947081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d87e43-27d7-4884-950c-818f72fbb504 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:24.947081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d87e43-27d7-4884-950c-818f72fbb504 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:24.947081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d87e43-27d7-4884-950c-818f72fbb504 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:24.947081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 87/170] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:24.952332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d50728-3f77-40cf-9702-8c572d15cbe3 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:24.960861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee736bd-df48-4d1f-8fef-bee860e03da4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:24.961144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee736bd-df48-4d1f-8fef-bee860e03da4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:24.961419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee736bd-df48-4d1f-8fef-bee860e03da4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:24.990581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-eee736bd-df48-4d1f-8fef-bee860e03da4 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:24.991769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-eee736bd-df48-4d1f-8fef-bee860e03da4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:25.009773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:25.122091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 88/171] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:25.152438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee736bd-df48-4d1f-8fef-bee860e03da4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:25.178529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d50728-3f77-40cf-9702-8c572d15cbe3 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:25.254240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:25.264541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6acb0ba-4af1-4a31-9a4d-8869209ce957 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:25.265162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6acb0ba-4af1-4a31-9a4d-8869209ce957 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:25.265441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6acb0ba-4af1-4a31-9a4d-8869209ce957 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:25.265717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6acb0ba-4af1-4a31-9a4d-8869209ce957 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:25.266393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6acb0ba-4af1-4a31-9a4d-8869209ce957 None admin] RBAC: Adding query filter params (name=member) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:25.266695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6acb0ba-4af1-4a31-9a4d-8869209ce957 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:25.268087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6acb0ba-4af1-4a31-9a4d-8869209ce957 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:25.291463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 84/172] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:25.296614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6acb0ba-4af1-4a31-9a4d-8869209ce957 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:25.336799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee736bd-df48-4d1f-8fef-bee860e03da4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:25.407861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090826dc-b632-4054-bf30-b131c324e042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:25.411380 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090826dc-b632-4054-bf30-b131c324e042 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:25.411887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090826dc-b632-4054-bf30-b131c324e042 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:25.412374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090826dc-b632-4054-bf30-b131c324e042 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:25.428526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:25.437500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090826dc-b632-4054-bf30-b131c324e042 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-090826dc-b632-4054-bf30-b131c324e042 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:25.442850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:25.443808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:25.444622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:25.451778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 89/173] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:25 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:22:25.453793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0edc0a-4098-4201-a822-5b9ff91592ec None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:25.458027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0edc0a-4098-4201-a822-5b9ff91592ec None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:25.458528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0edc0a-4098-4201-a822-5b9ff91592ec None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:25.460046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 90/174] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:25 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:25.466905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0edc0a-4098-4201-a822-5b9ff91592ec None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:25.503664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6acb0ba-4af1-4a31-9a4d-8869209ce957 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:25.541735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0196fc-2638-4161-b48a-ce722161bf8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:25.542393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0196fc-2638-4161-b48a-ce722161bf8d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:25.542714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0196fc-2638-4161-b48a-ce722161bf8d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:25.542995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0196fc-2638-4161-b48a-ce722161bf8d None admin] PATH_INFO: `/v3/users/18674c9f5ca2436b921d8f52a40444be` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:25.544134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0196fc-2638-4161-b48a-ce722161bf8d None admin] RBAC: Authorizing `identity:get_user(user_id=18674c9f5ca2436b921d8f52a40444be)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:25.545710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0196fc-2638-4161-b48a-ce722161bf8d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:25.547369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 85/175] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:25 2022] GET /identity/v3/users/18674c9f5ca2436b921d8f52a40444be => generated 275 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:25.548566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2567219-a1b9-4925-a926-72fa57dbd53b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:25.548847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2567219-a1b9-4925-a926-72fa57dbd53b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:25.549121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2567219-a1b9-4925-a926-72fa57dbd53b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:25.567662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-d2567219-a1b9-4925-a926-72fa57dbd53b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:25.568905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d2567219-a1b9-4925-a926-72fa57dbd53b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:25.607496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0edc0a-4098-4201-a822-5b9ff91592ec None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:25.633471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:25.647848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 86/176] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:25.653690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2567219-a1b9-4925-a926-72fa57dbd53b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:25.659332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f17f55-e751-42dc-8e5e-896c82225eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:25.660337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f17f55-e751-42dc-8e5e-896c82225eec None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:25.660931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f17f55-e751-42dc-8e5e-896c82225eec None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:25.662569 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f17f55-e751-42dc-8e5e-896c82225eec None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:25.663581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f17f55-e751-42dc-8e5e-896c82225eec None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:25.664065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f17f55-e751-42dc-8e5e-896c82225eec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:25.666590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f17f55-e751-42dc-8e5e-896c82225eec None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:25.683158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 91/177] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:25 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:25.684818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f17f55-e751-42dc-8e5e-896c82225eec None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:25.772731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:25.812396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2567219-a1b9-4925-a926-72fa57dbd53b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:25.814219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f17f55-e751-42dc-8e5e-896c82225eec None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:25.850710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d09addb-cafe-41d0-9559-c04836369676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:25.851607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d09addb-cafe-41d0-9559-c04836369676 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:25.852056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d09addb-cafe-41d0-9559-c04836369676 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:25.852523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d09addb-cafe-41d0-9559-c04836369676 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:25.860300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a37943-88ea-4c54-a15f-0e6a87d615a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:25.861326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a37943-88ea-4c54-a15f-0e6a87d615a0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:25.861611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a37943-88ea-4c54-a15f-0e6a87d615a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:25.861886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a37943-88ea-4c54-a15f-0e6a87d615a0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:25.864216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d09addb-cafe-41d0-9559-c04836369676 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-4d09addb-cafe-41d0-9559-c04836369676 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:25.869259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:25.870129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:25.870914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:25.878100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 87/178] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:22:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:25.879123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d09addb-cafe-41d0-9559-c04836369676 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:25.936152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:25.954122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a37943-88ea-4c54-a15f-0e6a87d615a0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:25.955867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a37943-88ea-4c54-a15f-0e6a87d615a0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:25.997290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d09addb-cafe-41d0-9559-c04836369676 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:26.023609 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:26.035068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61caae7-8166-4c63-b2a7-68bb2a8fa481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:26.038273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61caae7-8166-4c63-b2a7-68bb2a8fa481 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.038563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61caae7-8166-4c63-b2a7-68bb2a8fa481 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.038836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61caae7-8166-4c63-b2a7-68bb2a8fa481 None admin] PATH_INFO: `/v3/users/18674c9f5ca2436b921d8f52a40444be` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.040514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61caae7-8166-4c63-b2a7-68bb2a8fa481 None admin] RBAC: Authorizing `identity:get_user(user_id=18674c9f5ca2436b921d8f52a40444be)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:26.042355 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61caae7-8166-4c63-b2a7-68bb2a8fa481 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:26.044492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 88/179] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:25 2022] GET /identity/v3/users/18674c9f5ca2436b921d8f52a40444be => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:26.049059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3298cd82-d06e-461b-b45d-a2cce139bee6 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.049059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3298cd82-d06e-461b-b45d-a2cce139bee6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.049059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3298cd82-d06e-461b-b45d-a2cce139bee6 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.049059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 89/180] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:26.049059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3298cd82-d06e-461b-b45d-a2cce139bee6 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:26.075170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 92/181] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:26.075869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a37943-88ea-4c54-a15f-0e6a87d615a0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:26.176008 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3298cd82-d06e-461b-b45d-a2cce139bee6 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:26.177578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a37943-88ea-4c54-a15f-0e6a87d615a0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:26.195719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:26.215029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080b7c91-b209-4ad4-b043-c58f84a678ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:26.215719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080b7c91-b209-4ad4-b043-c58f84a678ae None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.216005 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080b7c91-b209-4ad4-b043-c58f84a678ae None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.216105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6eb303-6788-4186-b635-0e7648685ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:26.216281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080b7c91-b209-4ad4-b043-c58f84a678ae None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.216658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6eb303-6788-4186-b635-0e7648685ba0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.216933 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6eb303-6788-4186-b635-0e7648685ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.217028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080b7c91-b209-4ad4-b043-c58f84a678ae None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:26.217200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6eb303-6788-4186-b635-0e7648685ba0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.217294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080b7c91-b209-4ad4-b043-c58f84a678ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:26.218299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6eb303-6788-4186-b635-0e7648685ba0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:26.218595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080b7c91-b209-4ad4-b043-c58f84a678ae None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:26.219869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6eb303-6788-4186-b635-0e7648685ba0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:26.221660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 93/182] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:26.222419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6eb303-6788-4186-b635-0e7648685ba0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:26.232390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 90/183] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:26.234326 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d644e1ea-4dee-469a-bb7a-c3d878efac66 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.235148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d644e1ea-4dee-469a-bb7a-c3d878efac66 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.235634 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d644e1ea-4dee-469a-bb7a-c3d878efac66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.254232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:26.256882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-d644e1ea-4dee-469a-bb7a-c3d878efac66 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:26.258125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d644e1ea-4dee-469a-bb7a-c3d878efac66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:26.328653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6eb303-6788-4186-b635-0e7648685ba0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:26.335257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 91/184] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:26.336099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d644e1ea-4dee-469a-bb7a-c3d878efac66 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:26.375591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987f6110-ea2e-4333-81f7-36a40fceccf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:26.376712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987f6110-ea2e-4333-81f7-36a40fceccf7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.377206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987f6110-ea2e-4333-81f7-36a40fceccf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.377682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987f6110-ea2e-4333-81f7-36a40fceccf7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.452216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987f6110-ea2e-4333-81f7-36a40fceccf7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:26.453708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987f6110-ea2e-4333-81f7-36a40fceccf7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:26.457341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d644e1ea-4dee-469a-bb7a-c3d878efac66 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:26.483453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:26.486983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:26.500813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e91cef8-dbaf-44f2-8152-6cb2ca56ed6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:26.501821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e91cef8-dbaf-44f2-8152-6cb2ca56ed6c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.502295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e91cef8-dbaf-44f2-8152-6cb2ca56ed6c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.502746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e91cef8-dbaf-44f2-8152-6cb2ca56ed6c None admin] PATH_INFO: `/v3/projects/057afb8734e34b9687d7b36edb0034b4` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.504238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e91cef8-dbaf-44f2-8152-6cb2ca56ed6c None admin] RBAC: Authorizing `identity:get_project(project_id=057afb8734e34b9687d7b36edb0034b4)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:26.506034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e91cef8-dbaf-44f2-8152-6cb2ca56ed6c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:26.508397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 92/185] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:26 2022] GET /identity/v3/projects/057afb8734e34b9687d7b36edb0034b4 => generated 312 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:26.510017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e91cef8-dbaf-44f2-8152-6cb2ca56ed6c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:26.569996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 94/186] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:26.569996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987f6110-ea2e-4333-81f7-36a40fceccf7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:26.639879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e91cef8-dbaf-44f2-8152-6cb2ca56ed6c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:26.674758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987f6110-ea2e-4333-81f7-36a40fceccf7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:26.683276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961ed5bf-8236-41db-b565-ee77e0d5e8f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:26.684150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961ed5bf-8236-41db-b565-ee77e0d5e8f8 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.684648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961ed5bf-8236-41db-b565-ee77e0d5e8f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.685125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961ed5bf-8236-41db-b565-ee77e0d5e8f8 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.685840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961ed5bf-8236-41db-b565-ee77e0d5e8f8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:26.687296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961ed5bf-8236-41db-b565-ee77e0d5e8f8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:26.705795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:26.710036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 93/187] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:22:26 2022] POST /identity/v3/users => generated 297 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:26.712303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a9c886-d54f-474c-864c-fd1376db6f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:26.712979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a9c886-d54f-474c-864c-fd1376db6f18 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.713257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a9c886-d54f-474c-864c-fd1376db6f18 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.713386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961ed5bf-8236-41db-b565-ee77e0d5e8f8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:26.713555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a9c886-d54f-474c-864c-fd1376db6f18 None admin] PATH_INFO: `/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.730272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a9c886-d54f-474c-864c-fd1376db6f18 None admin] RBAC: Authorizing `identity:get_user(user_id=d3d8b2a89a0b4cf493b1fa014b74fc7d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:26.731821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a9c886-d54f-474c-864c-fd1376db6f18 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:26.733607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 95/188] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:26 2022] GET /identity/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d => generated 283 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:26.734398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a9c886-d54f-474c-864c-fd1376db6f18 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:26.761107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:26.844551 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961ed5bf-8236-41db-b565-ee77e0d5e8f8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:26.856154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a9c886-d54f-474c-864c-fd1376db6f18 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:26.887453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05939c3-af29-4f65-a98e-590d0dcf82f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:26.888069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05939c3-af29-4f65-a98e-590d0dcf82f6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.888372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05939c3-af29-4f65-a98e-590d0dcf82f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.888645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05939c3-af29-4f65-a98e-590d0dcf82f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.889408 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05939c3-af29-4f65-a98e-590d0dcf82f6 None admin] RBAC: Adding query filter params (role.id=27476172dbb84c6e86b9070e02d06100, scope.project.id=057afb8734e34b9687d7b36edb0034b4, user.id=18674c9f5ca2436b921d8f52a40444be) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:26.889717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05939c3-af29-4f65-a98e-590d0dcf82f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:26.891270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05939c3-af29-4f65-a98e-590d0dcf82f6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:26.897169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cbc97a-edd0-4f18-b61c-7800fc4e275e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:26.897981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cbc97a-edd0-4f18-b61c-7800fc4e275e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.904429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cbc97a-edd0-4f18-b61c-7800fc4e275e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.904622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 94/189] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:26 2022] GET /identity/v3/role_assignments?user.id=18674c9f5ca2436b921d8f52a40444be&scope.project.id=057afb8734e34b9687d7b36edb0034b4&role.id=27476172dbb84c6e86b9070e02d06100 => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:26.904742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cbc97a-edd0-4f18-b61c-7800fc4e275e None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.904836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05939c3-af29-4f65-a98e-590d0dcf82f6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:26.904919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cbc97a-edd0-4f18-b61c-7800fc4e275e None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:26.904919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cbc97a-edd0-4f18-b61c-7800fc4e275e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:26.904919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 96/190] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:26 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:26.904919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94366c9a-dbbe-4074-a8bf-0eceabdccadf None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:26.904919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94366c9a-dbbe-4074-a8bf-0eceabdccadf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:26.904919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94366c9a-dbbe-4074-a8bf-0eceabdccadf None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:26.906115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 97/191] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:26.906839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94366c9a-dbbe-4074-a8bf-0eceabdccadf None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:26.991329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:27.058418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:27.059852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94366c9a-dbbe-4074-a8bf-0eceabdccadf None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:27.062880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05939c3-af29-4f65-a98e-590d0dcf82f6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:27.113552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be8ec93-e710-4b1d-a7ad-dfe39f790aad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:27.114203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be8ec93-e710-4b1d-a7ad-dfe39f790aad None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:27.114481 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be8ec93-e710-4b1d-a7ad-dfe39f790aad None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:27.114755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be8ec93-e710-4b1d-a7ad-dfe39f790aad None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:27.122228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0de5b5a-f82a-40aa-a416-c177d6192c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:27.122847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0de5b5a-f82a-40aa-a416-c177d6192c8d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:27.123133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0de5b5a-f82a-40aa-a416-c177d6192c8d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:27.123397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0de5b5a-f82a-40aa-a416-c177d6192c8d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:27.127157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be8ec93-e710-4b1d-a7ad-dfe39f790aad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-7be8ec93-e710-4b1d-a7ad-dfe39f790aad None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:27.135750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:27.136805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:27.137595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:27.138194 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 95/192] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:27.138757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be8ec93-e710-4b1d-a7ad-dfe39f790aad None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:27.215961 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0de5b5a-f82a-40aa-a416-c177d6192c8d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:27.217963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0de5b5a-f82a-40aa-a416-c177d6192c8d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:27.268589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be8ec93-e710-4b1d-a7ad-dfe39f790aad None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:27.294147 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:27.323754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa60dec5-7147-4eaa-ae41-4de00a824e3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:27.324400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa60dec5-7147-4eaa-ae41-4de00a824e3f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:27.324705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa60dec5-7147-4eaa-ae41-4de00a824e3f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:27.325334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa60dec5-7147-4eaa-ae41-4de00a824e3f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:27.326597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa60dec5-7147-4eaa-ae41-4de00a824e3f None admin] RBAC: Adding query filter params (role.id=48389faedba2470a885fae19084bb456, scope.project.id=5f976380e1494db480dc80868d51a0bc, user.id=18674c9f5ca2436b921d8f52a40444be) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:27.326915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa60dec5-7147-4eaa-ae41-4de00a824e3f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:27.328955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa60dec5-7147-4eaa-ae41-4de00a824e3f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:27.337176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 96/193] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:27 2022] GET /identity/v3/role_assignments?user.id=18674c9f5ca2436b921d8f52a40444be&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=48389faedba2470a885fae19084bb456 => generated 608 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:27.338995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474a96fb-4fa7-47d6-b54f-c15d74b705a0 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:27.339565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474a96fb-4fa7-47d6-b54f-c15d74b705a0 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:27.339851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474a96fb-4fa7-47d6-b54f-c15d74b705a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:27.348969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 98/194] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:27.349685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0de5b5a-f82a-40aa-a416-c177d6192c8d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:27.376828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-474a96fb-4fa7-47d6-b54f-c15d74b705a0 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:27.378045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-474a96fb-4fa7-47d6-b54f-c15d74b705a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:27.409004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:27.461786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 97/195] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:27.463527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474a96fb-4fa7-47d6-b54f-c15d74b705a0 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:27.567203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0de5b5a-f82a-40aa-a416-c177d6192c8d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:27.607700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0092c551-d316-490f-95ff-49ffa0489e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:27.608298 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0092c551-d316-490f-95ff-49ffa0489e7b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:27.608623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0092c551-d316-490f-95ff-49ffa0489e7b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:27.608890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0092c551-d316-490f-95ff-49ffa0489e7b None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:27.609601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0092c551-d316-490f-95ff-49ffa0489e7b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:27.609909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0092c551-d316-490f-95ff-49ffa0489e7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:27.611288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0092c551-d316-490f-95ff-49ffa0489e7b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:27.615225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474a96fb-4fa7-47d6-b54f-c15d74b705a0 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:27.621116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 99/196] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:27.626632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0092c551-d316-490f-95ff-49ffa0489e7b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:27.654153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2010db-449a-4836-93e8-90e4abd647e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:27.655065 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2010db-449a-4836-93e8-90e4abd647e6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:27.655490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2010db-449a-4836-93e8-90e4abd647e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:27.655893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2010db-449a-4836-93e8-90e4abd647e6 None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:27.657208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2010db-449a-4836-93e8-90e4abd647e6 None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:27.658810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2010db-449a-4836-93e8-90e4abd647e6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:27.660846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 98/197] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:27 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:27.664501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:27.667268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2010db-449a-4836-93e8-90e4abd647e6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:27.690925 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:27.775026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2010db-449a-4836-93e8-90e4abd647e6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:27.811638 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efcdb9d-ad3e-4c34-a9ea-e04ef7592ace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:27.812621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efcdb9d-ad3e-4c34-a9ea-e04ef7592ace None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:27.813117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efcdb9d-ad3e-4c34-a9ea-e04ef7592ace None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:27.813582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efcdb9d-ad3e-4c34-a9ea-e04ef7592ace None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:27.814528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efcdb9d-ad3e-4c34-a9ea-e04ef7592ace None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3, user.id=d3d8b2a89a0b4cf493b1fa014b74fc7d) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:27.815103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efcdb9d-ad3e-4c34-a9ea-e04ef7592ace None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:27.816984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efcdb9d-ad3e-4c34-a9ea-e04ef7592ace None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:27.823855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 99/198] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:27 2022] GET /identity/v3/role_assignments?user.id=d3d8b2a89a0b4cf493b1fa014b74fc7d&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:27.835011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0092c551-d316-490f-95ff-49ffa0489e7b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:27.857552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efcdb9d-ad3e-4c34-a9ea-e04ef7592ace None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:27.898969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3980f9db-4f15-42b8-87f3-4ad9646d9ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:27.900125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3980f9db-4f15-42b8-87f3-4ad9646d9ff0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:27.900797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3980f9db-4f15-42b8-87f3-4ad9646d9ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:27.901278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3980f9db-4f15-42b8-87f3-4ad9646d9ff0 None admin] PATH_INFO: `/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:27.903011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3980f9db-4f15-42b8-87f3-4ad9646d9ff0 None admin] RBAC: Authorizing `identity:get_user(user_id=d3d8b2a89a0b4cf493b1fa014b74fc7d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:27.906374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3980f9db-4f15-42b8-87f3-4ad9646d9ff0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:27.909459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 100/199] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:27 2022] GET /identity/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d => generated 283 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:27.918905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3980f9db-4f15-42b8-87f3-4ad9646d9ff0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:28.036231 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:28.064064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efcdb9d-ad3e-4c34-a9ea-e04ef7592ace None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:28.070955 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:28.093799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3980f9db-4f15-42b8-87f3-4ad9646d9ff0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:28.119853 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef069238-6d99-4231-b3f4-1808439f1a25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:28.120552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef069238-6d99-4231-b3f4-1808439f1a25 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:28.120830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef069238-6d99-4231-b3f4-1808439f1a25 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:28.121100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef069238-6d99-4231-b3f4-1808439f1a25 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:28.143118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef069238-6d99-4231-b3f4-1808439f1a25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ef069238-6d99-4231-b3f4-1808439f1a25 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:28.154783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:28.155700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:28.156533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 100/200] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:28.162015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef069238-6d99-4231-b3f4-1808439f1a25 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:28.168063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27425958-7f7a-4c9e-b24f-6aaddf4fbe16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:28.168698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27425958-7f7a-4c9e-b24f-6aaddf4fbe16 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:28.168974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27425958-7f7a-4c9e-b24f-6aaddf4fbe16 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:28.175837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27425958-7f7a-4c9e-b24f-6aaddf4fbe16 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:28.302851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef069238-6d99-4231-b3f4-1808439f1a25 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:28.317358 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27425958-7f7a-4c9e-b24f-6aaddf4fbe16 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:28.318862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27425958-7f7a-4c9e-b24f-6aaddf4fbe16 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:28.341388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491f9493-159e-42b0-88ca-c3f24e741e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:28.342000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491f9493-159e-42b0-88ca-c3f24e741e21 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:28.342276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491f9493-159e-42b0-88ca-c3f24e741e21 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:28.342547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491f9493-159e-42b0-88ca-c3f24e741e21 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:28.343206 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491f9493-159e-42b0-88ca-c3f24e741e21 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:28.343507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491f9493-159e-42b0-88ca-c3f24e741e21 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:28.344969 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491f9493-159e-42b0-88ca-c3f24e741e21 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:28.354777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 101/201] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:28 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:28.363557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491f9493-159e-42b0-88ca-c3f24e741e21 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:28.379110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:28.469811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:28.499461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 101/202] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 581 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:28.504913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27425958-7f7a-4c9e-b24f-6aaddf4fbe16 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:28.524954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491f9493-159e-42b0-88ca-c3f24e741e21 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:28.562114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec42486d-6ea8-48c6-9e67-bd4dc535fdcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:28.562749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec42486d-6ea8-48c6-9e67-bd4dc535fdcc None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:28.563028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec42486d-6ea8-48c6-9e67-bd4dc535fdcc None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:28.563302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec42486d-6ea8-48c6-9e67-bd4dc535fdcc None admin] PATH_INFO: `/v3/users/07c216b73cfd4842a37c9f2bb1e2051d` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:28.564484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec42486d-6ea8-48c6-9e67-bd4dc535fdcc None admin] RBAC: Authorizing `identity:get_user(user_id=07c216b73cfd4842a37c9f2bb1e2051d)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:28.566001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec42486d-6ea8-48c6-9e67-bd4dc535fdcc None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:28.572304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 102/203] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:28 2022] GET /identity/v3/users/07c216b73cfd4842a37c9f2bb1e2051d => generated 289 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:28.573495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec42486d-6ea8-48c6-9e67-bd4dc535fdcc None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:28.666771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:28.677632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27425958-7f7a-4c9e-b24f-6aaddf4fbe16 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:28.717943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec42486d-6ea8-48c6-9e67-bd4dc535fdcc None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:28.742064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e9b05a-a47c-4ee1-ac92-a585f203d8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:28.742695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e9b05a-a47c-4ee1-ac92-a585f203d8dd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:28.742975 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e9b05a-a47c-4ee1-ac92-a585f203d8dd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:28.743235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e9b05a-a47c-4ee1-ac92-a585f203d8dd None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:28.749035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e9b05a-a47c-4ee1-ac92-a585f203d8dd None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:28.750495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e9b05a-a47c-4ee1-ac92-a585f203d8dd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:28.752459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 102/204] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:28 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:28.767626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e9b05a-a47c-4ee1-ac92-a585f203d8dd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:28.771490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ce7c34-6c30-4a7d-af34-ea70cfc59f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:28.772099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ce7c34-6c30-4a7d-af34-ea70cfc59f18 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:28.772400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ce7c34-6c30-4a7d-af34-ea70cfc59f18 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:28.772674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ce7c34-6c30-4a7d-af34-ea70cfc59f18 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:28.848536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ce7c34-6c30-4a7d-af34-ea70cfc59f18 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:28.850003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ce7c34-6c30-4a7d-af34-ea70cfc59f18 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:28.884706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:28.933330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:28.985166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 103/205] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:28.988817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e9b05a-a47c-4ee1-ac92-a585f203d8dd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:28.991139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ce7c34-6c30-4a7d-af34-ea70cfc59f18 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:29.094415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436f93cf-9c64-47ef-b764-dab65326282f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:29.095691 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436f93cf-9c64-47ef-b764-dab65326282f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:29.096168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436f93cf-9c64-47ef-b764-dab65326282f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:29.096684 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436f93cf-9c64-47ef-b764-dab65326282f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:29.097641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436f93cf-9c64-47ef-b764-dab65326282f None admin] RBAC: Adding query filter params (role.id=48389faedba2470a885fae19084bb456, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3, user.id=d3d8b2a89a0b4cf493b1fa014b74fc7d) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:29.098145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436f93cf-9c64-47ef-b764-dab65326282f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:29.099789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436f93cf-9c64-47ef-b764-dab65326282f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:29.108499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ce7c34-6c30-4a7d-af34-ea70cfc59f18 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:29.112904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 103/206] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:28 2022] GET /identity/v3/role_assignments?user.id=d3d8b2a89a0b4cf493b1fa014b74fc7d&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=48389faedba2470a885fae19084bb456 => generated 259 bytes in 346 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:29.148670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cb7adb-6c61-4451-b771-5128a5ad2b73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:29.149309 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb7adb-6c61-4451-b771-5128a5ad2b73 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:29.149613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb7adb-6c61-4451-b771-5128a5ad2b73 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:29.149929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb7adb-6c61-4451-b771-5128a5ad2b73 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:29.151381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cb7adb-6c61-4451-b771-5128a5ad2b73 None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:29.152873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cb7adb-6c61-4451-b771-5128a5ad2b73 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:29.154545 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 104/207] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:28 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:29.160693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436f93cf-9c64-47ef-b764-dab65326282f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:29.307714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436f93cf-9c64-47ef-b764-dab65326282f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:29.326784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:29.345852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7952d074-b3c7-4bf5-9fc8-fa8aa04bbd1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:29.346476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7952d074-b3c7-4bf5-9fc8-fa8aa04bbd1b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:29.346757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7952d074-b3c7-4bf5-9fc8-fa8aa04bbd1b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:29.347099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7952d074-b3c7-4bf5-9fc8-fa8aa04bbd1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:29.347842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7952d074-b3c7-4bf5-9fc8-fa8aa04bbd1b None admin] RBAC: Adding query filter params (role.id=8922a4a1fc6e4b22815abb10e1f19807, scope.project.id=5f976380e1494db480dc80868d51a0bc, user.id=07c216b73cfd4842a37c9f2bb1e2051d) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:29.348151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7952d074-b3c7-4bf5-9fc8-fa8aa04bbd1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:29.349755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7952d074-b3c7-4bf5-9fc8-fa8aa04bbd1b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:29.356634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 104/208] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:29 2022] GET /identity/v3/role_assignments?user.id=07c216b73cfd4842a37c9f2bb1e2051d&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=8922a4a1fc6e4b22815abb10e1f19807 => generated 608 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:29.658277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e2002-8688-4760-bdc4-d53d5231daa7 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:29.658769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e2002-8688-4760-bdc4-d53d5231daa7 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:29.658769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e2002-8688-4760-bdc4-d53d5231daa7 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:29.660054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 105/209] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:29.667745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42fe1aa-fc26-4961-b2cb-90ce7678b6af None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:29.669344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbc8b5e-9f68-491a-9fd9-08255a72420f None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:29.669834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbc8b5e-9f68-491a-9fd9-08255a72420f None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:29.670323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbc8b5e-9f68-491a-9fd9-08255a72420f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:29.670796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42fe1aa-fc26-4961-b2cb-90ce7678b6af None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:29.673892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42fe1aa-fc26-4961-b2cb-90ce7678b6af None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:29.675694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 105/210] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:29 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:29.686491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478878bb-a69f-451e-bfa3-3c763beab1ed None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:29.687059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478878bb-a69f-451e-bfa3-3c763beab1ed None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:29.687605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478878bb-a69f-451e-bfa3-3c763beab1ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:29.693534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-cdbc8b5e-9f68-491a-9fd9-08255a72420f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:29.694728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-cdbc8b5e-9f68-491a-9fd9-08255a72420f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:29.708815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-478878bb-a69f-451e-bfa3-3c763beab1ed None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:29.710424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-478878bb-a69f-451e-bfa3-3c763beab1ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:29.740709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:29.786120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 106/211] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:29.804674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 106/212] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:30.067228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbc8b5e-9f68-491a-9fd9-08255a72420f None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:30.093536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478878bb-a69f-451e-bfa3-3c763beab1ed None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:30.205477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbc8b5e-9f68-491a-9fd9-08255a72420f None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:30.207303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478878bb-a69f-451e-bfa3-3c763beab1ed None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:30.223384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:30.246918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d251d53-71a4-4ce8-83ea-1c25f1df452e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:30.247553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d251d53-71a4-4ce8-83ea-1c25f1df452e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:30.247834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d251d53-71a4-4ce8-83ea-1c25f1df452e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:30.248109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d251d53-71a4-4ce8-83ea-1c25f1df452e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:30.249587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d251d53-71a4-4ce8-83ea-1c25f1df452e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:30.251089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d251d53-71a4-4ce8-83ea-1c25f1df452e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:30.252880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 107/213] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:30.254982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcc1ecf-030b-4b76-bea6-87c84dd0494c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:30.255837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcc1ecf-030b-4b76-bea6-87c84dd0494c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:30.256312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcc1ecf-030b-4b76-bea6-87c84dd0494c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:30.256809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcc1ecf-030b-4b76-bea6-87c84dd0494c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:30.258089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d251d53-71a4-4ce8-83ea-1c25f1df452e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:30.270788 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcc1ecf-030b-4b76-bea6-87c84dd0494c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-afcc1ecf-030b-4b76-bea6-87c84dd0494c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:30.278620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:30.279524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:30.280322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:30.281610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 107/214] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:30.288294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcc1ecf-030b-4b76-bea6-87c84dd0494c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:30.334844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:30.382171 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d251d53-71a4-4ce8-83ea-1c25f1df452e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:30.409876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcc1ecf-030b-4b76-bea6-87c84dd0494c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:30.419808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03be63f9-f9f2-4b34-9f74-9fe06590b191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:30.420666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03be63f9-f9f2-4b34-9f74-9fe06590b191 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:30.420946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03be63f9-f9f2-4b34-9f74-9fe06590b191 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:30.421216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03be63f9-f9f2-4b34-9f74-9fe06590b191 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:30.421873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03be63f9-f9f2-4b34-9f74-9fe06590b191 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:30.423315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03be63f9-f9f2-4b34-9f74-9fe06590b191 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:30.447338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c958d2e4-b256-4036-b996-bbdc5c344119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:30.447957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c958d2e4-b256-4036-b996-bbdc5c344119 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:30.448229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c958d2e4-b256-4036-b996-bbdc5c344119 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:30.448531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c958d2e4-b256-4036-b996-bbdc5c344119 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:30.449220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c958d2e4-b256-4036-b996-bbdc5c344119 None admin] RBAC: Adding query filter params (name=member) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:30.449611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c958d2e4-b256-4036-b996-bbdc5c344119 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:30.451216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c958d2e4-b256-4036-b996-bbdc5c344119 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:30.452119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 108/215] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:22:30 2022] POST /identity/v3/users => generated 297 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:30.453209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21303b2d-666e-4ccc-87cc-ba36536edf40 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:30.453497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21303b2d-666e-4ccc-87cc-ba36536edf40 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:30.453772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21303b2d-666e-4ccc-87cc-ba36536edf40 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:30.455237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 109/216] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:30.463670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fbce7d-efb7-4d63-a032-0d9ed4ed2da3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:30.463670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fbce7d-efb7-4d63-a032-0d9ed4ed2da3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:30.463670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fbce7d-efb7-4d63-a032-0d9ed4ed2da3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:30.468021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 108/217] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:30.473708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c958d2e4-b256-4036-b996-bbdc5c344119 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:30.484288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-30fbce7d-efb7-4d63-a032-0d9ed4ed2da3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:30.485958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-30fbce7d-efb7-4d63-a032-0d9ed4ed2da3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:30.521445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:30.571096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 110/218] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:30.601447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:30.609153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c958d2e4-b256-4036-b996-bbdc5c344119 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:30.646670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c05502-d744-4db1-b26b-3c30f837d42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:30.647295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c05502-d744-4db1-b26b-3c30f837d42d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:30.647568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c05502-d744-4db1-b26b-3c30f837d42d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:30.647835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c05502-d744-4db1-b26b-3c30f837d42d None admin] PATH_INFO: `/v3/users/3aaf745d371e4fe7883ccfe92d2cbc5c` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:30.664095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c05502-d744-4db1-b26b-3c30f837d42d None admin] RBAC: Authorizing `identity:get_user(user_id=3aaf745d371e4fe7883ccfe92d2cbc5c)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:30.665859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c05502-d744-4db1-b26b-3c30f837d42d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:30.667953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 109/219] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:30 2022] GET /identity/v3/users/3aaf745d371e4fe7883ccfe92d2cbc5c => generated 297 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:30.673542 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fbce7d-efb7-4d63-a032-0d9ed4ed2da3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:30.780734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fbce7d-efb7-4d63-a032-0d9ed4ed2da3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:30.816934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6b28e4-4df0-42df-b891-f7f9806d611d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:30.817868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6b28e4-4df0-42df-b891-f7f9806d611d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:30.818339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6b28e4-4df0-42df-b891-f7f9806d611d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:30.818835 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6b28e4-4df0-42df-b891-f7f9806d611d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:30.855224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:30.861507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c05502-d744-4db1-b26b-3c30f837d42d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:30.902820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6b28e4-4df0-42df-b891-f7f9806d611d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:30.904295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6b28e4-4df0-42df-b891-f7f9806d611d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:31.008085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c05502-d744-4db1-b26b-3c30f837d42d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:31.035052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 111/220] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:31.041812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6b28e4-4df0-42df-b891-f7f9806d611d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:31.070125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab87c07c-6c1a-4314-bebe-2d14967d14eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:31.071067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab87c07c-6c1a-4314-bebe-2d14967d14eb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:31.073243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab87c07c-6c1a-4314-bebe-2d14967d14eb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:31.073809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab87c07c-6c1a-4314-bebe-2d14967d14eb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:31.096602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab87c07c-6c1a-4314-bebe-2d14967d14eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ab87c07c-6c1a-4314-bebe-2d14967d14eb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:31.103458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:31.104416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:31.105276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:31.113589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 110/221] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:22:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:22:31.122764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab87c07c-6c1a-4314-bebe-2d14967d14eb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:31.148732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:31.189424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:31.238252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6b28e4-4df0-42df-b891-f7f9806d611d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:31.294271 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f189a573-9003-4536-b3d3-c0659f67c0ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:31.294915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189a573-9003-4536-b3d3-c0659f67c0ef None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:31.295288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189a573-9003-4536-b3d3-c0659f67c0ef None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:31.295463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189a573-9003-4536-b3d3-c0659f67c0ef None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:31.296665 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f189a573-9003-4536-b3d3-c0659f67c0ef None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:31.300680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab87c07c-6c1a-4314-bebe-2d14967d14eb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:31.300820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f189a573-9003-4536-b3d3-c0659f67c0ef None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:31.300820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 112/222] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:31 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:31.307401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f189a573-9003-4536-b3d3-c0659f67c0ef None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:31.378695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d15458-2746-4055-a7e4-5ed2775b7159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:31.379600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d15458-2746-4055-a7e4-5ed2775b7159 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:31.380062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d15458-2746-4055-a7e4-5ed2775b7159 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:31.380557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d15458-2746-4055-a7e4-5ed2775b7159 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:31.381405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d15458-2746-4055-a7e4-5ed2775b7159 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:31.382211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d15458-2746-4055-a7e4-5ed2775b7159 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:31.384717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d15458-2746-4055-a7e4-5ed2775b7159 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:31.401644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 111/223] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:31.408145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d15458-2746-4055-a7e4-5ed2775b7159 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:31.490887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f189a573-9003-4536-b3d3-c0659f67c0ef None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:31.553244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:31.565947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88bde38-9d2e-41b0-9135-f2788f235537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:31.566560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88bde38-9d2e-41b0-9135-f2788f235537 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:31.566838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88bde38-9d2e-41b0-9135-f2788f235537 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:31.567115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88bde38-9d2e-41b0-9135-f2788f235537 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:31.567854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88bde38-9d2e-41b0-9135-f2788f235537 None admin] RBAC: Adding query filter params (role.id=27476172dbb84c6e86b9070e02d06100, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3, user.id=3aaf745d371e4fe7883ccfe92d2cbc5c) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:31.568161 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88bde38-9d2e-41b0-9135-f2788f235537 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:31.569640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88bde38-9d2e-41b0-9135-f2788f235537 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:31.574498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:31.583316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 113/224] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:31 2022] GET /identity/v3/role_assignments?user.id=3aaf745d371e4fe7883ccfe92d2cbc5c&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=27476172dbb84c6e86b9070e02d06100 => generated 259 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:31.601851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d15458-2746-4055-a7e4-5ed2775b7159 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:31.616500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e388fe4-52ad-4742-ae71-54736ea2e703 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:31.617117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e388fe4-52ad-4742-ae71-54736ea2e703 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:31.617596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e388fe4-52ad-4742-ae71-54736ea2e703 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:31.620072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 114/225] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:22:31.625770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff955be4-959f-44fb-b76d-9d177d3cd4c2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:31.626371 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff955be4-959f-44fb-b76d-9d177d3cd4c2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:31.626864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff955be4-959f-44fb-b76d-9d177d3cd4c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:31.646035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ff955be4-959f-44fb-b76d-9d177d3cd4c2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:31.647629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ff955be4-959f-44fb-b76d-9d177d3cd4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:31.682429 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78bf534-9263-4d46-8600-041c1db9ef53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:31.683089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78bf534-9263-4d46-8600-041c1db9ef53 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:31.683336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78bf534-9263-4d46-8600-041c1db9ef53 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:31.683653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78bf534-9263-4d46-8600-041c1db9ef53 None admin] PATH_INFO: `/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:31.689317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78bf534-9263-4d46-8600-041c1db9ef53 None admin] RBAC: Authorizing `identity:get_user(user_id=d3d8b2a89a0b4cf493b1fa014b74fc7d)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:31.690828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78bf534-9263-4d46-8600-041c1db9ef53 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:31.698951 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 112/226] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:31 2022] GET /identity/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d => generated 283 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:31.700279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78bf534-9263-4d46-8600-041c1db9ef53 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:31.783273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 115/227] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:31.871212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78bf534-9263-4d46-8600-041c1db9ef53 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:31.907467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df918eff-eb7d-4987-9e6f-5d42a2969511 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:31.908074 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df918eff-eb7d-4987-9e6f-5d42a2969511 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:31.908373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df918eff-eb7d-4987-9e6f-5d42a2969511 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:31.908701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df918eff-eb7d-4987-9e6f-5d42a2969511 None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:31.909839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df918eff-eb7d-4987-9e6f-5d42a2969511 None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:31.911326 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df918eff-eb7d-4987-9e6f-5d42a2969511 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:31.913308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 113/228] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:31 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:31.919216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff955be4-959f-44fb-b76d-9d177d3cd4c2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:32.055411 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff955be4-959f-44fb-b76d-9d177d3cd4c2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:32.094625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:32.104445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd6918d-9070-4b52-9bf1-e1baa97d3a2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:32.105359 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd6918d-9070-4b52-9bf1-e1baa97d3a2d None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:32.105874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd6918d-9070-4b52-9bf1-e1baa97d3a2d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:32.106350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd6918d-9070-4b52-9bf1-e1baa97d3a2d None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:32.109414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd6918d-9070-4b52-9bf1-e1baa97d3a2d None admin] RBAC: Authorizing `identity:create_grant(project_id=06f515d57f2143c8ad79c6d8440ceab3, user_id=d3d8b2a89a0b4cf493b1fa014b74fc7d, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:32.111139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd6918d-9070-4b52-9bf1-e1baa97d3a2d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:32.136240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 116/229] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:22:31 2022] PUT /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:32.146275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df918eff-eb7d-4987-9e6f-5d42a2969511 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:32.165657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:32.296542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df918eff-eb7d-4987-9e6f-5d42a2969511 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:32.335808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4eea344-b04c-49bb-a458-30521bd4833b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:32.336471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eea344-b04c-49bb-a458-30521bd4833b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:32.336755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eea344-b04c-49bb-a458-30521bd4833b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:32.337060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eea344-b04c-49bb-a458-30521bd4833b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:32.348592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4eea344-b04c-49bb-a458-30521bd4833b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-b4eea344-b04c-49bb-a458-30521bd4833b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:32.355400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:32.356287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:32.357109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 114/230] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:32.362512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd6918d-9070-4b52-9bf1-e1baa97d3a2d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:32.501277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd6918d-9070-4b52-9bf1-e1baa97d3a2d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:32.537301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c613687e-fe28-40ca-af39-ee8c53820398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:32.537905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c613687e-fe28-40ca-af39-ee8c53820398 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:32.538180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c613687e-fe28-40ca-af39-ee8c53820398 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:32.538450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c613687e-fe28-40ca-af39-ee8c53820398 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:32.539111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c613687e-fe28-40ca-af39-ee8c53820398 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:32.539409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c613687e-fe28-40ca-af39-ee8c53820398 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:32.541980 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c613687e-fe28-40ca-af39-ee8c53820398 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:32.554689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 117/231] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:32.560177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4eea344-b04c-49bb-a458-30521bd4833b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:32.653679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:32.687478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4eea344-b04c-49bb-a458-30521bd4833b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:32.722344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb21868-ff49-4634-a226-aa8f9f716123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:32.722978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb21868-ff49-4634-a226-aa8f9f716123 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:32.723256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb21868-ff49-4634-a226-aa8f9f716123 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:32.723528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb21868-ff49-4634-a226-aa8f9f716123 None admin] PATH_INFO: `/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:32.724761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb21868-ff49-4634-a226-aa8f9f716123 None admin] RBAC: Authorizing `identity:get_user(user_id=d3d8b2a89a0b4cf493b1fa014b74fc7d)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:32.726325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb21868-ff49-4634-a226-aa8f9f716123 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:32.728890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 115/232] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:32 2022] GET /identity/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d => generated 283 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:32.734106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c613687e-fe28-40ca-af39-ee8c53820398 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:32.818936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:32.895640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c613687e-fe28-40ca-af39-ee8c53820398 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:32.947832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5925bf3-9d4a-4b15-a051-373c26f4ca4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:32.948673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5925bf3-9d4a-4b15-a051-373c26f4ca4b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:32.948962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5925bf3-9d4a-4b15-a051-373c26f4ca4b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:32.949237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5925bf3-9d4a-4b15-a051-373c26f4ca4b None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:32.952539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5925bf3-9d4a-4b15-a051-373c26f4ca4b None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:32.954105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5925bf3-9d4a-4b15-a051-373c26f4ca4b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:32.956329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 118/233] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:32 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:32.961949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb21868-ff49-4634-a226-aa8f9f716123 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:33.047138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feec117-7b72-4af8-8e7b-ea4cd03e4be8 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:33.047450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feec117-7b72-4af8-8e7b-ea4cd03e4be8 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:33.047692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3feec117-7b72-4af8-8e7b-ea4cd03e4be8 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:33.049693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 119/234] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:33.058269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6d0b52-0d7a-46b9-a1a0-9cccc3a3ac81 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:33.058917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6d0b52-0d7a-46b9-a1a0-9cccc3a3ac81 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:33.059462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6d0b52-0d7a-46b9-a1a0-9cccc3a3ac81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:33.080872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9c6d0b52-0d7a-46b9-a1a0-9cccc3a3ac81 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:33.082377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9c6d0b52-0d7a-46b9-a1a0-9cccc3a3ac81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:33.123669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb21868-ff49-4634-a226-aa8f9f716123 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:33.159969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 120/235] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:33.176978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aa654a-badc-4d38-8aa3-37dcea739a9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:33.177650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aa654a-badc-4d38-8aa3-37dcea739a9a None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:33.177934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aa654a-badc-4d38-8aa3-37dcea739a9a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:33.178211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aa654a-badc-4d38-8aa3-37dcea739a9a None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d/roles/48389faedba2470a885fae19084bb456` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:33.180592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aa654a-badc-4d38-8aa3-37dcea739a9a None admin] RBAC: Authorizing `identity:create_grant(project_id=06f515d57f2143c8ad79c6d8440ceab3, user_id=d3d8b2a89a0b4cf493b1fa014b74fc7d, role_id=48389faedba2470a885fae19084bb456)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:33.182154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aa654a-badc-4d38-8aa3-37dcea739a9a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:33.200575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:33.206503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 116/236] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:22:32 2022] PUT /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d/roles/48389faedba2470a885fae19084bb456 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:33.449299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6d0b52-0d7a-46b9-a1a0-9cccc3a3ac81 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:33.508286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:33.565766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6d0b52-0d7a-46b9-a1a0-9cccc3a3ac81 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:33.579691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe88cea-68c1-4d89-9769-a1edb894529b None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:33.579985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe88cea-68c1-4d89-9769-a1edb894529b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:33.580691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe88cea-68c1-4d89-9769-a1edb894529b None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:33.582076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 117/237] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:33.597034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca332ad-9140-4eb1-9028-f985c4e2ff21 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:33.597321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca332ad-9140-4eb1-9028-f985c4e2ff21 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:33.597600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca332ad-9140-4eb1-9028-f985c4e2ff21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:33.616384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7ca332ad-9140-4eb1-9028-f985c4e2ff21 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:33.617834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7ca332ad-9140-4eb1-9028-f985c4e2ff21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:33.620044 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d6021e-c117-4e7d-ac7c-bff3385afbcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:33.620718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d6021e-c117-4e7d-ac7c-bff3385afbcb None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:33.621002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d6021e-c117-4e7d-ac7c-bff3385afbcb None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:33.621277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d6021e-c117-4e7d-ac7c-bff3385afbcb None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:33.643212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d6021e-c117-4e7d-ac7c-bff3385afbcb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f8d6021e-c117-4e7d-ac7c-bff3385afbcb None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:33.648158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:33.649060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:33.651174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 121/238] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:33 2022] GET /identity/v3/roles/reader => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:33.661346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d6021e-c117-4e7d-ac7c-bff3385afbcb None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:33.712782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 118/239] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:33.823722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d6021e-c117-4e7d-ac7c-bff3385afbcb None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:33.834220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:33.873746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dc5a1c-ce1b-416c-aa8f-16fb6a38507c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:33.874356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dc5a1c-ce1b-416c-aa8f-16fb6a38507c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:33.874632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dc5a1c-ce1b-416c-aa8f-16fb6a38507c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:33.874901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dc5a1c-ce1b-416c-aa8f-16fb6a38507c None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:33.875569 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dc5a1c-ce1b-416c-aa8f-16fb6a38507c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:33.875871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dc5a1c-ce1b-416c-aa8f-16fb6a38507c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:33.877291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dc5a1c-ce1b-416c-aa8f-16fb6a38507c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:33.886826 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 122/240] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:33.893084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca332ad-9140-4eb1-9028-f985c4e2ff21 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:34.001456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca332ad-9140-4eb1-9028-f985c4e2ff21 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:34.005445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dc5a1c-ce1b-416c-aa8f-16fb6a38507c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:34.057768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:34.075675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99afa3f8-24e1-4633-926f-2386ceba3df4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:34.076313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99afa3f8-24e1-4633-926f-2386ceba3df4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:34.076625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99afa3f8-24e1-4633-926f-2386ceba3df4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:34.076892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99afa3f8-24e1-4633-926f-2386ceba3df4 None admin] PATH_INFO: `/v3/users/70b8e04c7aa346fd87334f8d292f31f0` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:34.104263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99afa3f8-24e1-4633-926f-2386ceba3df4 None admin] RBAC: Authorizing `identity:get_user(user_id=70b8e04c7aa346fd87334f8d292f31f0)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:34.105751 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99afa3f8-24e1-4633-926f-2386ceba3df4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:34.111799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 119/241] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:33 2022] GET /identity/v3/users/70b8e04c7aa346fd87334f8d292f31f0 => generated 297 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:34.118917 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99afa3f8-24e1-4633-926f-2386ceba3df4 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:34.172616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dc5a1c-ce1b-416c-aa8f-16fb6a38507c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:34.208712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caebe5ea-16d2-4ea7-bb6a-1daf6425acf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:34.209347 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caebe5ea-16d2-4ea7-bb6a-1daf6425acf2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:34.209626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caebe5ea-16d2-4ea7-bb6a-1daf6425acf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:34.209895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caebe5ea-16d2-4ea7-bb6a-1daf6425acf2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:34.211028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caebe5ea-16d2-4ea7-bb6a-1daf6425acf2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:34.213351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caebe5ea-16d2-4ea7-bb6a-1daf6425acf2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:34.215117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 123/242] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:34.220971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caebe5ea-16d2-4ea7-bb6a-1daf6425acf2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:34.264780 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:34.335783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caebe5ea-16d2-4ea7-bb6a-1daf6425acf2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:34.371720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a78c14-1bc3-41d6-8018-50313ec66025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:34.372330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a78c14-1bc3-41d6-8018-50313ec66025 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:34.372645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a78c14-1bc3-41d6-8018-50313ec66025 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:34.372914 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a78c14-1bc3-41d6-8018-50313ec66025 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:34.373578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a78c14-1bc3-41d6-8018-50313ec66025 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:34.375135 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a78c14-1bc3-41d6-8018-50313ec66025 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:34.405965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 124/243] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:22:34 2022] POST /identity/v3/users => generated 293 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:34.422710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99afa3f8-24e1-4633-926f-2386ceba3df4 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:34.470315 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4022b90-593c-49df-9ed6-43e1564f9bf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:34.470933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4022b90-593c-49df-9ed6-43e1564f9bf1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:34.471207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4022b90-593c-49df-9ed6-43e1564f9bf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:34.471479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4022b90-593c-49df-9ed6-43e1564f9bf1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:34.536845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:34.567329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4022b90-593c-49df-9ed6-43e1564f9bf1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:34.568866 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4022b90-593c-49df-9ed6-43e1564f9bf1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:34.719554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 120/244] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 602 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:34.729370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a78c14-1bc3-41d6-8018-50313ec66025 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:34.742680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b52f51-661a-4cc1-aef9-86e5dae99154 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:34.742859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b52f51-661a-4cc1-aef9-86e5dae99154 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:34.743060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b52f51-661a-4cc1-aef9-86e5dae99154 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:34.744514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 121/245] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:34.749776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483309b0-6f8d-4e8e-a4eb-030d88ef7b91 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:34.750269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483309b0-6f8d-4e8e-a4eb-030d88ef7b91 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:34.750683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483309b0-6f8d-4e8e-a4eb-030d88ef7b91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:34.776806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-483309b0-6f8d-4e8e-a4eb-030d88ef7b91 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:34.778037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-483309b0-6f8d-4e8e-a4eb-030d88ef7b91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:34.856491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:34.870682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a78c14-1bc3-41d6-8018-50313ec66025 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:34.871921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 122/246] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:34.908771 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7778bfc-b9a1-4523-b5ed-71868452808b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:34.909402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7778bfc-b9a1-4523-b5ed-71868452808b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:34.909679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7778bfc-b9a1-4523-b5ed-71868452808b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:34.909950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7778bfc-b9a1-4523-b5ed-71868452808b None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:34.911126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7778bfc-b9a1-4523-b5ed-71868452808b None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:34.912604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7778bfc-b9a1-4523-b5ed-71868452808b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:34.914314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 125/247] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:34 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:34.920668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483309b0-6f8d-4e8e-a4eb-030d88ef7b91 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:34.956751 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:34.987303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926d00c-dd6e-4369-8570-08eab31073c1 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:34.987603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926d00c-dd6e-4369-8570-08eab31073c1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:34.987842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926d00c-dd6e-4369-8570-08eab31073c1 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:34.989593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 126/248] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:22:34.997849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcf09a9-2e9e-46ba-be12-ef17549512fa None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:34.998122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcf09a9-2e9e-46ba-be12-ef17549512fa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:34.998386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcf09a9-2e9e-46ba-be12-ef17549512fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:35.018749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ddcf09a9-2e9e-46ba-be12-ef17549512fa None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:35.020745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ddcf09a9-2e9e-46ba-be12-ef17549512fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:35.030824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483309b0-6f8d-4e8e-a4eb-030d88ef7b91 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:35.078793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c94150-656d-4055-a9c5-cf1eb20f1e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:35.079437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c94150-656d-4055-a9c5-cf1eb20f1e92 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:35.079723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c94150-656d-4055-a9c5-cf1eb20f1e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:35.079998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c94150-656d-4055-a9c5-cf1eb20f1e92 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:35.081013 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c94150-656d-4055-a9c5-cf1eb20f1e92 None admin] RBAC: Adding query filter params (role.id=8922a4a1fc6e4b22815abb10e1f19807, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3, user.id=70b8e04c7aa346fd87334f8d292f31f0) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:35.081323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c94150-656d-4055-a9c5-cf1eb20f1e92 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:35.082764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c94150-656d-4055-a9c5-cf1eb20f1e92 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:35.092010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 123/249] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:34 2022] GET /identity/v3/role_assignments?user.id=70b8e04c7aa346fd87334f8d292f31f0&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=8922a4a1fc6e4b22815abb10e1f19807 => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:35.133623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 127/250] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:35.186181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c94150-656d-4055-a9c5-cf1eb20f1e92 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:35.284559 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:35.307723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c94150-656d-4055-a9c5-cf1eb20f1e92 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:35.346345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373e346d-2e62-413b-836c-0abbbcd845e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:35.347358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373e346d-2e62-413b-836c-0abbbcd845e3 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:35.347856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373e346d-2e62-413b-836c-0abbbcd845e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:35.348366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373e346d-2e62-413b-836c-0abbbcd845e3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:35.359650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373e346d-2e62-413b-836c-0abbbcd845e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-373e346d-2e62-413b-836c-0abbbcd845e3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:35.365328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:35.366310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:35.367104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:35.370449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 124/251] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:35.375875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcf09a9-2e9e-46ba-be12-ef17549512fa None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:35.507313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:35.510155 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373e346d-2e62-413b-836c-0abbbcd845e3 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:35.519793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcf09a9-2e9e-46ba-be12-ef17549512fa None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:35.564626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e708aca4-76a6-4c02-be9c-800f041edad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:35.565254 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e708aca4-76a6-4c02-be9c-800f041edad0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:35.565531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e708aca4-76a6-4c02-be9c-800f041edad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:35.565804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e708aca4-76a6-4c02-be9c-800f041edad0 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:35.566478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e708aca4-76a6-4c02-be9c-800f041edad0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:35.566779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e708aca4-76a6-4c02-be9c-800f041edad0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:35.568174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e708aca4-76a6-4c02-be9c-800f041edad0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:35.582766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 128/252] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:35.589117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e708aca4-76a6-4c02-be9c-800f041edad0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:35.752076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373e346d-2e62-413b-836c-0abbbcd845e3 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:35.798679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e708aca4-76a6-4c02-be9c-800f041edad0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:35.825230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bf31b1-8f87-469a-a365-be47fed77275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:35.827811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bf31b1-8f87-469a-a365-be47fed77275 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:35.828654 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bf31b1-8f87-469a-a365-be47fed77275 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:35.829385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bf31b1-8f87-469a-a365-be47fed77275 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:35.857905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bf31b1-8f87-469a-a365-be47fed77275 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-95bf31b1-8f87-469a-a365-be47fed77275 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:35.864292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:35.865225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:35.866040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:35.872552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:35.874264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 125/253] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:22:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 366 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:35.877811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bf31b1-8f87-469a-a365-be47fed77275 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:35.881278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88067080-cfa8-43ed-912c-7140aca7fa49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:35.881917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88067080-cfa8-43ed-912c-7140aca7fa49 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:35.882190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88067080-cfa8-43ed-912c-7140aca7fa49 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:35.882458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88067080-cfa8-43ed-912c-7140aca7fa49 None admin] PATH_INFO: `/v3/users/3aaf745d371e4fe7883ccfe92d2cbc5c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:35.885984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88067080-cfa8-43ed-912c-7140aca7fa49 None admin] RBAC: Authorizing `identity:get_user(user_id=3aaf745d371e4fe7883ccfe92d2cbc5c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:35.887440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88067080-cfa8-43ed-912c-7140aca7fa49 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:35.889172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 129/254] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:35 2022] GET /identity/v3/users/3aaf745d371e4fe7883ccfe92d2cbc5c => generated 297 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:35.895025 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88067080-cfa8-43ed-912c-7140aca7fa49 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:35.939451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:36.021241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bf31b1-8f87-469a-a365-be47fed77275 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:36.064977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b2fd42-c89f-4c06-83c7-35e4bfc75ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:36.065739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b2fd42-c89f-4c06-83c7-35e4bfc75ded None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:36.066114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b2fd42-c89f-4c06-83c7-35e4bfc75ded None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:36.066484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b2fd42-c89f-4c06-83c7-35e4bfc75ded None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:36.067252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b2fd42-c89f-4c06-83c7-35e4bfc75ded None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:36.067656 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b2fd42-c89f-4c06-83c7-35e4bfc75ded None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:36.069175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b2fd42-c89f-4c06-83c7-35e4bfc75ded None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:36.084939 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 126/255] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:36.086149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f96159-9d42-442c-92b3-23029b814913 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:36.088251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f96159-9d42-442c-92b3-23029b814913 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:36.088648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f96159-9d42-442c-92b3-23029b814913 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:36.089870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 127/256] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:36.092863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f96159-9d42-442c-92b3-23029b814913 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:36.110309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88067080-cfa8-43ed-912c-7140aca7fa49 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:36.175563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607cc87b-70f4-4069-9aba-ecc7d25e84b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:36.176200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607cc87b-70f4-4069-9aba-ecc7d25e84b0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:36.176508 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607cc87b-70f4-4069-9aba-ecc7d25e84b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:36.177917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607cc87b-70f4-4069-9aba-ecc7d25e84b0 None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:36.179905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607cc87b-70f4-4069-9aba-ecc7d25e84b0 None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:36.181556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607cc87b-70f4-4069-9aba-ecc7d25e84b0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:36.187765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 130/257] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:35 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:36.191149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9db8e4-7bcc-4e86-8a9b-bb201ff3dc94 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:36.191446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9db8e4-7bcc-4e86-8a9b-bb201ff3dc94 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:36.191718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9db8e4-7bcc-4e86-8a9b-bb201ff3dc94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:36.228728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:36.236703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-df9db8e4-7bcc-4e86-8a9b-bb201ff3dc94 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:36.237900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-df9db8e4-7bcc-4e86-8a9b-bb201ff3dc94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:36.298526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f96159-9d42-442c-92b3-23029b814913 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:36.364330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccda8686-c41a-4cfe-80ff-7521feff5323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:36.365902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccda8686-c41a-4cfe-80ff-7521feff5323 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:36.366185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccda8686-c41a-4cfe-80ff-7521feff5323 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:36.366456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccda8686-c41a-4cfe-80ff-7521feff5323 None admin] PATH_INFO: `/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:36.368103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccda8686-c41a-4cfe-80ff-7521feff5323 None admin] RBAC: Authorizing `identity:get_user(user_id=d3d8b2a89a0b4cf493b1fa014b74fc7d)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:36.369983 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccda8686-c41a-4cfe-80ff-7521feff5323 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:36.371878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 128/258] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:36 2022] GET /identity/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d => generated 283 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:36.375530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 131/259] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:36.376435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccda8686-c41a-4cfe-80ff-7521feff5323 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:36.378096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9db8e4-7bcc-4e86-8a9b-bb201ff3dc94 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:36.428728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:36.566477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9db8e4-7bcc-4e86-8a9b-bb201ff3dc94 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:36.609098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7e8d5b-2196-46c7-a17d-1abde5b6f2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:36.610109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7e8d5b-2196-46c7-a17d-1abde5b6f2d2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:36.610392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7e8d5b-2196-46c7-a17d-1abde5b6f2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:36.610662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7e8d5b-2196-46c7-a17d-1abde5b6f2d2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:36.614727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccda8686-c41a-4cfe-80ff-7521feff5323 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:36.694723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:36.707268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7e8d5b-2196-46c7-a17d-1abde5b6f2d2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:36.708965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7e8d5b-2196-46c7-a17d-1abde5b6f2d2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:36.713852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789037ba-2ef6-466d-a553-6921750064b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:36.714762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789037ba-2ef6-466d-a553-6921750064b2 None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:36.715245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789037ba-2ef6-466d-a553-6921750064b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:36.715735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789037ba-2ef6-466d-a553-6921750064b2 None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/users/3aaf745d371e4fe7883ccfe92d2cbc5c/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:36.725129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789037ba-2ef6-466d-a553-6921750064b2 None admin] RBAC: Authorizing `identity:create_grant(project_id=06f515d57f2143c8ad79c6d8440ceab3, user_id=3aaf745d371e4fe7883ccfe92d2cbc5c, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:36.726864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789037ba-2ef6-466d-a553-6921750064b2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:36.731826 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:36.764374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 129/260] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:22:36 2022] PUT /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/users/3aaf745d371e4fe7883ccfe92d2cbc5c/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:36.765823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789037ba-2ef6-466d-a553-6921750064b2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:36.879082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 132/261] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 501 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:36.886932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7e8d5b-2196-46c7-a17d-1abde5b6f2d2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:36.980128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789037ba-2ef6-466d-a553-6921750064b2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:37.039426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ff5132-6f47-4720-911e-e9307739e9d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:37.040098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ff5132-6f47-4720-911e-e9307739e9d2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:37.044575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ff5132-6f47-4720-911e-e9307739e9d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:37.044575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ff5132-6f47-4720-911e-e9307739e9d2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:37.072795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ff5132-6f47-4720-911e-e9307739e9d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-30ff5132-6f47-4720-911e-e9307739e9d2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:37.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:37.089251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:37.090054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 130/262] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:37.097910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ff5132-6f47-4720-911e-e9307739e9d2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:37.115591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7e8d5b-2196-46c7-a17d-1abde5b6f2d2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:37.166506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:37.170255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:37.199833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbf0af1-f4be-49ff-83d7-308d48791930 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:37.200502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbf0af1-f4be-49ff-83d7-308d48791930 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:37.200784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbf0af1-f4be-49ff-83d7-308d48791930 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:37.201057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbf0af1-f4be-49ff-83d7-308d48791930 None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:37.207916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbf0af1-f4be-49ff-83d7-308d48791930 None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:37.209484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbf0af1-f4be-49ff-83d7-308d48791930 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:37.211463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 133/263] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:36 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:37.218724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbf0af1-f4be-49ff-83d7-308d48791930 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:37.262801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ff5132-6f47-4720-911e-e9307739e9d2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:37.323983 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a915d4c7-d534-4241-807b-3ed52467c821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:37.324789 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a915d4c7-d534-4241-807b-3ed52467c821 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:37.325121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a915d4c7-d534-4241-807b-3ed52467c821 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:37.325402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a915d4c7-d534-4241-807b-3ed52467c821 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:37.326076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a915d4c7-d534-4241-807b-3ed52467c821 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:37.326382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a915d4c7-d534-4241-807b-3ed52467c821 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:37.327771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a915d4c7-d534-4241-807b-3ed52467c821 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:37.343451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 131/264] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:37.349461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a915d4c7-d534-4241-807b-3ed52467c821 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:37.470333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbf0af1-f4be-49ff-83d7-308d48791930 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:37.516656 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a915d4c7-d534-4241-807b-3ed52467c821 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:37.531941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:37.557994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5cded3-ca47-4863-b3d8-0d8bd443684f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:37.558620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5cded3-ca47-4863-b3d8-0d8bd443684f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:37.558899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5cded3-ca47-4863-b3d8-0d8bd443684f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:37.559173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5cded3-ca47-4863-b3d8-0d8bd443684f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:37.559911 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5cded3-ca47-4863-b3d8-0d8bd443684f None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3, user.id=d3d8b2a89a0b4cf493b1fa014b74fc7d) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:37.560230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5cded3-ca47-4863-b3d8-0d8bd443684f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:37.561872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5cded3-ca47-4863-b3d8-0d8bd443684f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:37.562665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecec0ed9-c0a0-412d-a7f2-7c549abf17d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:37.563490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecec0ed9-c0a0-412d-a7f2-7c549abf17d0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:37.563978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecec0ed9-c0a0-412d-a7f2-7c549abf17d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:37.564420 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecec0ed9-c0a0-412d-a7f2-7c549abf17d0 None admin] PATH_INFO: `/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:37.568466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 134/265] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:37 2022] GET /identity/v3/role_assignments?user.id=d3d8b2a89a0b4cf493b1fa014b74fc7d&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 608 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:37.569430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecec0ed9-c0a0-412d-a7f2-7c549abf17d0 None admin] RBAC: Authorizing `identity:get_user(user_id=d3d8b2a89a0b4cf493b1fa014b74fc7d)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:37.571066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecec0ed9-c0a0-412d-a7f2-7c549abf17d0 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:37.573880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 132/266] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:37 2022] GET /identity/v3/users/d3d8b2a89a0b4cf493b1fa014b74fc7d => generated 283 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:37.579951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5cded3-ca47-4863-b3d8-0d8bd443684f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:37.675646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:37.724926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5cded3-ca47-4863-b3d8-0d8bd443684f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:37.763931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f673ff4-8a7b-4a5c-90b8-26f1fb4ca4b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:37.764569 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f673ff4-8a7b-4a5c-90b8-26f1fb4ca4b1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:37.764847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f673ff4-8a7b-4a5c-90b8-26f1fb4ca4b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:37.765122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f673ff4-8a7b-4a5c-90b8-26f1fb4ca4b1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:37.847781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f673ff4-8a7b-4a5c-90b8-26f1fb4ca4b1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:37.849285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f673ff4-8a7b-4a5c-90b8-26f1fb4ca4b1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:37.858050 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabf5ec2-d887-4e2c-9b27-0243fd670f08 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:37.858977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabf5ec2-d887-4e2c-9b27-0243fd670f08 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:37.859465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabf5ec2-d887-4e2c-9b27-0243fd670f08 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:37.864551 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 133/267] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:37.870034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18bea5e-eced-45ce-acfd-2a91d05723ab None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:37.870678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18bea5e-eced-45ce-acfd-2a91d05723ab None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:37.871161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18bea5e-eced-45ce-acfd-2a91d05723ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:37.905426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b18bea5e-eced-45ce-acfd-2a91d05723ab None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:37.907101 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b18bea5e-eced-45ce-acfd-2a91d05723ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:37.933514 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:38.007073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 135/268] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:38.016380 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f673ff4-8a7b-4a5c-90b8-26f1fb4ca4b1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:38.050072 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 134/269] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:38.134145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f673ff4-8a7b-4a5c-90b8-26f1fb4ca4b1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:38.173265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed36279-1017-4749-88b2-f20ad0e5989c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:38.174252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed36279-1017-4749-88b2-f20ad0e5989c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:38.174707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed36279-1017-4749-88b2-f20ad0e5989c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:38.175154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed36279-1017-4749-88b2-f20ad0e5989c None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:38.176546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed36279-1017-4749-88b2-f20ad0e5989c None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:38.178233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed36279-1017-4749-88b2-f20ad0e5989c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:38.180156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 136/270] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:38 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:38.187128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18bea5e-eced-45ce-acfd-2a91d05723ab None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:38.299179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bccf1e7-2f4d-4228-98e7-202ceaf3946f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:38.299731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bccf1e7-2f4d-4228-98e7-202ceaf3946f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:38.300200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bccf1e7-2f4d-4228-98e7-202ceaf3946f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:38.301799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 137/271] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:22:38.309852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b33a4bb-0ea1-49d2-9a86-a3d3ebb6ed68 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:38.310419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b33a4bb-0ea1-49d2-9a86-a3d3ebb6ed68 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:38.310863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b33a4bb-0ea1-49d2-9a86-a3d3ebb6ed68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:38.335313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18bea5e-eced-45ce-acfd-2a91d05723ab None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:38.338583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9b33a4bb-0ea1-49d2-9a86-a3d3ebb6ed68 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:38.339765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9b33a4bb-0ea1-49d2-9a86-a3d3ebb6ed68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:38.350623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:38.380215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dda1af-38b1-42ed-aded-4a49142df6a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:38.381162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dda1af-38b1-42ed-aded-4a49142df6a5 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:38.381670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dda1af-38b1-42ed-aded-4a49142df6a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:38.382154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dda1af-38b1-42ed-aded-4a49142df6a5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:38.383146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:38.383402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dda1af-38b1-42ed-aded-4a49142df6a5 None admin] RBAC: Adding query filter params (role.id=48389faedba2470a885fae19084bb456, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3, user.id=d3d8b2a89a0b4cf493b1fa014b74fc7d) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:38.383903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dda1af-38b1-42ed-aded-4a49142df6a5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:38.386262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dda1af-38b1-42ed-aded-4a49142df6a5 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:38.394591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 135/272] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:38 2022] GET /identity/v3/role_assignments?user.id=d3d8b2a89a0b4cf493b1fa014b74fc7d&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=48389faedba2470a885fae19084bb456 => generated 608 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:38.419743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e12daa8-5c0c-4f68-b8dc-12219b8da0f6 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:38.420630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e12daa8-5c0c-4f68-b8dc-12219b8da0f6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:38.421110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e12daa8-5c0c-4f68-b8dc-12219b8da0f6 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:38.423161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 136/273] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:22:38.429147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d8af67-e11d-44f3-9459-79c7fd90b09b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:38.430055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d8af67-e11d-44f3-9459-79c7fd90b09b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:38.430531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d8af67-e11d-44f3-9459-79c7fd90b09b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:38.453559 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-81d8af67-e11d-44f3-9459-79c7fd90b09b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:38.455157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-81d8af67-e11d-44f3-9459-79c7fd90b09b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:38.493136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 138/274] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:38.514988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b33a4bb-0ea1-49d2-9a86-a3d3ebb6ed68 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:38.582291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 137/275] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:38.628828 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b33a4bb-0ea1-49d2-9a86-a3d3ebb6ed68 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:38.669037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29531f9-1e1a-4e87-a06b-db883ecb2265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:38.670052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29531f9-1e1a-4e87-a06b-db883ecb2265 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:38.670529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29531f9-1e1a-4e87-a06b-db883ecb2265 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:38.671059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29531f9-1e1a-4e87-a06b-db883ecb2265 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:38.682570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29531f9-1e1a-4e87-a06b-db883ecb2265 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f29531f9-1e1a-4e87-a06b-db883ecb2265 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:38.688041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:38.688957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:38.689773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:38.692440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 139/276] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:38.699075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d8af67-e11d-44f3-9459-79c7fd90b09b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:38.814035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d8af67-e11d-44f3-9459-79c7fd90b09b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:38.816364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:38.833452 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29531f9-1e1a-4e87-a06b-db883ecb2265 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:38.852323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f763ff0d-6a5a-4695-a1c3-40ab93767895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:38.853024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f763ff0d-6a5a-4695-a1c3-40ab93767895 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:38.853300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f763ff0d-6a5a-4695-a1c3-40ab93767895 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:38.853571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f763ff0d-6a5a-4695-a1c3-40ab93767895 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:38.854426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f763ff0d-6a5a-4695-a1c3-40ab93767895 None admin] RBAC: Adding query filter params (name=member) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:38.854728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f763ff0d-6a5a-4695-a1c3-40ab93767895 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:38.854942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:38.856098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f763ff0d-6a5a-4695-a1c3-40ab93767895 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:38.865425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 138/277] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:38.871591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f763ff0d-6a5a-4695-a1c3-40ab93767895 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:38.976841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29531f9-1e1a-4e87-a06b-db883ecb2265 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:38.982564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f763ff0d-6a5a-4695-a1c3-40ab93767895 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:39.015495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e86090-28cd-4f1b-94f7-aabc6f846179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:39.016138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e86090-28cd-4f1b-94f7-aabc6f846179 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.016407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e86090-28cd-4f1b-94f7-aabc6f846179 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.016671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e86090-28cd-4f1b-94f7-aabc6f846179 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.017250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bd0785-cc3e-4c30-bd57-5bb1f0c0c439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:39.017870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bd0785-cc3e-4c30-bd57-5bb1f0c0c439 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.018149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bd0785-cc3e-4c30-bd57-5bb1f0c0c439 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.018431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bd0785-cc3e-4c30-bd57-5bb1f0c0c439 None admin] PATH_INFO: `/v3/users/037ab5c772cb4a42a03d3ae398497701` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.028857 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e86090-28cd-4f1b-94f7-aabc6f846179 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:39.032208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bd0785-cc3e-4c30-bd57-5bb1f0c0c439 None admin] RBAC: Authorizing `identity:get_user(user_id=037ab5c772cb4a42a03d3ae398497701)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-11e86090-28cd-4f1b-94f7-aabc6f846179 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:39.033674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:39.034614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:39.036046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:39.036754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bd0785-cc3e-4c30-bd57-5bb1f0c0c439 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:39.036825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 140/278] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:38 2022] GET /identity/v3/roles/reader => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:22:39.036825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e86090-28cd-4f1b-94f7-aabc6f846179 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:39.044048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 139/279] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:38 2022] GET /identity/v3/users/037ab5c772cb4a42a03d3ae398497701 => generated 293 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:39.047494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bd0785-cc3e-4c30-bd57-5bb1f0c0c439 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:39.077520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:39.116759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:39.151398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e86090-28cd-4f1b-94f7-aabc6f846179 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:39.160252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bd0785-cc3e-4c30-bd57-5bb1f0c0c439 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:39.188773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9370b05-2872-40d5-9d09-9a2441cc26a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:39.189413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9370b05-2872-40d5-9d09-9a2441cc26a7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.189694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9370b05-2872-40d5-9d09-9a2441cc26a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.189972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9370b05-2872-40d5-9d09-9a2441cc26a7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.191093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9370b05-2872-40d5-9d09-9a2441cc26a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:39.192588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9370b05-2872-40d5-9d09-9a2441cc26a7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:39.194153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 141/280] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:39.194921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9370b05-2872-40d5-9d09-9a2441cc26a7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:39.207820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494b4ed8-31a7-4d35-9cb8-1eff2116003f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:39.209040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494b4ed8-31a7-4d35-9cb8-1eff2116003f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.209498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494b4ed8-31a7-4d35-9cb8-1eff2116003f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.210073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494b4ed8-31a7-4d35-9cb8-1eff2116003f None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.211030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494b4ed8-31a7-4d35-9cb8-1eff2116003f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:39.211508 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494b4ed8-31a7-4d35-9cb8-1eff2116003f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:39.213374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494b4ed8-31a7-4d35-9cb8-1eff2116003f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:39.224608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 140/281] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:39 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:39.225716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494b4ed8-31a7-4d35-9cb8-1eff2116003f None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:39.303963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9370b05-2872-40d5-9d09-9a2441cc26a7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:39.319469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:39.336982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494b4ed8-31a7-4d35-9cb8-1eff2116003f None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:39.341042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dc93d8-f673-4e25-95f7-9ea63ffd39fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:39.341865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dc93d8-f673-4e25-95f7-9ea63ffd39fd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.342314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dc93d8-f673-4e25-95f7-9ea63ffd39fd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.342749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dc93d8-f673-4e25-95f7-9ea63ffd39fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.343640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dc93d8-f673-4e25-95f7-9ea63ffd39fd None admin] RBAC: Adding query filter params (role.id=27476172dbb84c6e86b9070e02d06100, user.id=037ab5c772cb4a42a03d3ae398497701, scope.system=all) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:39.344118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dc93d8-f673-4e25-95f7-9ea63ffd39fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:39.345876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dc93d8-f673-4e25-95f7-9ea63ffd39fd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:39.351399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 142/282] 10.222.0.25 () {56 vars in 1308 bytes} [Mon Sep 26 01:22:39 2022] GET /identity/v3/role_assignments?user.id=037ab5c772cb4a42a03d3ae398497701&scope.system=all&role.id=27476172dbb84c6e86b9070e02d06100 => generated 226 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:39.352562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dc93d8-f673-4e25-95f7-9ea63ffd39fd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:39.363562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:39.386992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a974e6-194e-4e99-9e27-5b600bf70ed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:39.387623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a974e6-194e-4e99-9e27-5b600bf70ed7 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.387910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a974e6-194e-4e99-9e27-5b600bf70ed7 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.388189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a974e6-194e-4e99-9e27-5b600bf70ed7 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.388911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a974e6-194e-4e99-9e27-5b600bf70ed7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:39.390410 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a974e6-194e-4e99-9e27-5b600bf70ed7 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:39.419753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 141/283] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:22:39 2022] POST /identity/v3/users => generated 293 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:39.510757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dc93d8-f673-4e25-95f7-9ea63ffd39fd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:39.556318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b925b2-e3d4-4b0a-902c-9c65f9616fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:39.557167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b925b2-e3d4-4b0a-902c-9c65f9616fff None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.557454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b925b2-e3d4-4b0a-902c-9c65f9616fff None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.557729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b925b2-e3d4-4b0a-902c-9c65f9616fff None admin] PATH_INFO: `/v3/users/70b8e04c7aa346fd87334f8d292f31f0` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.559074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b925b2-e3d4-4b0a-902c-9c65f9616fff None admin] RBAC: Authorizing `identity:get_user(user_id=70b8e04c7aa346fd87334f8d292f31f0)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:39.560728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b925b2-e3d4-4b0a-902c-9c65f9616fff None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:39.562872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 143/284] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:39 2022] GET /identity/v3/users/70b8e04c7aa346fd87334f8d292f31f0 => generated 297 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:39.573637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a974e6-194e-4e99-9e27-5b600bf70ed7 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:39.677293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2553af2e-8c84-4d80-9cc4-7f97b1935a77 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.677504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2553af2e-8c84-4d80-9cc4-7f97b1935a77 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.677812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2553af2e-8c84-4d80-9cc4-7f97b1935a77 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.679204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 144/285] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:39.684591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dffb2d-e7b7-4ba0-9f06-996d72f40cc6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.684877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dffb2d-e7b7-4ba0-9f06-996d72f40cc6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.685220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dffb2d-e7b7-4ba0-9f06-996d72f40cc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.690786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a974e6-194e-4e99-9e27-5b600bf70ed7 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:39.709531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a9dffb2d-e7b7-4ba0-9f06-996d72f40cc6 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:39.710834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a9dffb2d-e7b7-4ba0-9f06-996d72f40cc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:39.731090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea68231-ca9d-4255-a253-b9fb5c6833ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:39.731734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea68231-ca9d-4255-a253-b9fb5c6833ca None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.732015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea68231-ca9d-4255-a253-b9fb5c6833ca None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.732290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea68231-ca9d-4255-a253-b9fb5c6833ca None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.733513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea68231-ca9d-4255-a253-b9fb5c6833ca None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:39.735198 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea68231-ca9d-4255-a253-b9fb5c6833ca None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:39.736880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 142/286] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:39 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:39.744200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea68231-ca9d-4255-a253-b9fb5c6833ca None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:39.760540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:39.798046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:39.808694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 145/287] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:39.871031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea68231-ca9d-4255-a253-b9fb5c6833ca None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:39.906459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62ec1ee-70a7-4b07-8e13-128dae5f8fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:39.907109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62ec1ee-70a7-4b07-8e13-128dae5f8fe1 None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:39.907386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62ec1ee-70a7-4b07-8e13-128dae5f8fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:39.907656 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62ec1ee-70a7-4b07-8e13-128dae5f8fe1 None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/users/70b8e04c7aa346fd87334f8d292f31f0/roles/8922a4a1fc6e4b22815abb10e1f19807` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:39.909725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62ec1ee-70a7-4b07-8e13-128dae5f8fe1 None admin] RBAC: Authorizing `identity:create_grant(project_id=06f515d57f2143c8ad79c6d8440ceab3, user_id=70b8e04c7aa346fd87334f8d292f31f0, role_id=8922a4a1fc6e4b22815abb10e1f19807)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:39.911171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62ec1ee-70a7-4b07-8e13-128dae5f8fe1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:39.926288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 143/288] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:22:39 2022] PUT /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/users/70b8e04c7aa346fd87334f8d292f31f0/roles/8922a4a1fc6e4b22815abb10e1f19807 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:22:40.115760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dffb2d-e7b7-4ba0-9f06-996d72f40cc6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:40.229177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dffb2d-e7b7-4ba0-9f06-996d72f40cc6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:40.273321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b648f0c9-7812-4548-ad65-3226c39f6119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:40.273957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648f0c9-7812-4548-ad65-3226c39f6119 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:40.274238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648f0c9-7812-4548-ad65-3226c39f6119 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:40.274512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648f0c9-7812-4548-ad65-3226c39f6119 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:40.293260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b648f0c9-7812-4548-ad65-3226c39f6119 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b648f0c9-7812-4548-ad65-3226c39f6119 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:40.298192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:40.299096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:40.300288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 146/289] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:22:40.307776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62ec1ee-70a7-4b07-8e13-128dae5f8fe1 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:40.404113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:40.447064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62ec1ee-70a7-4b07-8e13-128dae5f8fe1 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:40.487585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a514b4c-673d-43da-abee-05589436a490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:40.488309 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a514b4c-673d-43da-abee-05589436a490 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:40.488618 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a514b4c-673d-43da-abee-05589436a490 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:40.488895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a514b4c-673d-43da-abee-05589436a490 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:40.489573 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a514b4c-673d-43da-abee-05589436a490 None admin] RBAC: Adding query filter params (name=member) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:40.489879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a514b4c-673d-43da-abee-05589436a490 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:40.491273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a514b4c-673d-43da-abee-05589436a490 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:40.500979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 144/290] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:40.506756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b648f0c9-7812-4548-ad65-3226c39f6119 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:40.605850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:40.671456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b648f0c9-7812-4548-ad65-3226c39f6119 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:40.717167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff66b533-ffc4-48a9-95b0-8fe6ab59588d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:40.717812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff66b533-ffc4-48a9-95b0-8fe6ab59588d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:40.718093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff66b533-ffc4-48a9-95b0-8fe6ab59588d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:40.718370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff66b533-ffc4-48a9-95b0-8fe6ab59588d None admin] PATH_INFO: `/v3/users/3aaf745d371e4fe7883ccfe92d2cbc5c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:40.719530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff66b533-ffc4-48a9-95b0-8fe6ab59588d None admin] RBAC: Authorizing `identity:get_user(user_id=3aaf745d371e4fe7883ccfe92d2cbc5c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:40.721046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff66b533-ffc4-48a9-95b0-8fe6ab59588d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:40.722635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 147/291] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:40 2022] GET /identity/v3/users/3aaf745d371e4fe7883ccfe92d2cbc5c => generated 297 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:40.729306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a514b4c-673d-43da-abee-05589436a490 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:40.849900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a514b4c-673d-43da-abee-05589436a490 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:40.882704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:40.887296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cced2f85-973a-4619-9de9-c3544b0e32f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:40.888112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cced2f85-973a-4619-9de9-c3544b0e32f0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:40.888535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cced2f85-973a-4619-9de9-c3544b0e32f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:40.888924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cced2f85-973a-4619-9de9-c3544b0e32f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:40.989880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cced2f85-973a-4619-9de9-c3544b0e32f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:40.991356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cced2f85-973a-4619-9de9-c3544b0e32f0 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:41.138034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 145/292] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:41.143805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff66b533-ffc4-48a9-95b0-8fe6ab59588d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:41.302392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff66b533-ffc4-48a9-95b0-8fe6ab59588d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:41.339551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:41.353256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09fcd13-0bd7-4660-ade4-428983065cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:41.354019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09fcd13-0bd7-4660-ade4-428983065cfc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:41.354304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09fcd13-0bd7-4660-ade4-428983065cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:41.354577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09fcd13-0bd7-4660-ade4-428983065cfc None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:41.355830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09fcd13-0bd7-4660-ade4-428983065cfc None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:41.357368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09fcd13-0bd7-4660-ade4-428983065cfc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:41.359391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 148/293] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:41 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:41.365313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cced2f85-973a-4619-9de9-c3544b0e32f0 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:41.425497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:41.521037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cced2f85-973a-4619-9de9-c3544b0e32f0 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:41.574625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db85d423-1e0e-4a0a-8e77-49663c439233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:41.575234 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db85d423-1e0e-4a0a-8e77-49663c439233 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:41.575515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db85d423-1e0e-4a0a-8e77-49663c439233 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:41.575791 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db85d423-1e0e-4a0a-8e77-49663c439233 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:41.576543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db85d423-1e0e-4a0a-8e77-49663c439233 None admin] RBAC: Adding query filter params (role.id=27476172dbb84c6e86b9070e02d06100, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3, user.id=3aaf745d371e4fe7883ccfe92d2cbc5c) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:41.576854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db85d423-1e0e-4a0a-8e77-49663c439233 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:41.578291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db85d423-1e0e-4a0a-8e77-49663c439233 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:41.587891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 146/294] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:41 2022] GET /identity/v3/role_assignments?user.id=3aaf745d371e4fe7883ccfe92d2cbc5c&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=27476172dbb84c6e86b9070e02d06100 => generated 608 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:41.859796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43344d-79bb-46ce-9efc-8c1ca3f46980 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:41.860191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43344d-79bb-46ce-9efc-8c1ca3f46980 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:41.860397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43344d-79bb-46ce-9efc-8c1ca3f46980 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:41.862128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 149/295] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:41.878447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa042a14-9446-426e-8b27-c8d0dd0a7ee9 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:41.878750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa042a14-9446-426e-8b27-c8d0dd0a7ee9 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:41.879001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa042a14-9446-426e-8b27-c8d0dd0a7ee9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:41.901160 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-aa042a14-9446-426e-8b27-c8d0dd0a7ee9 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:41.902393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-aa042a14-9446-426e-8b27-c8d0dd0a7ee9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:41.994479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 147/296] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:42.290047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ced56c3-b34f-4b38-9434-5633da3b8a86 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:42.291352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ced56c3-b34f-4b38-9434-5633da3b8a86 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:42.291352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ced56c3-b34f-4b38-9434-5633da3b8a86 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:42.292778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 150/297] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:42.296895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa042a14-9446-426e-8b27-c8d0dd0a7ee9 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:42.302243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f8532-64ae-47f7-a556-6799e4146567 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:42.302527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f8532-64ae-47f7-a556-6799e4146567 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:42.302798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f8532-64ae-47f7-a556-6799e4146567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:42.323821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-867f8532-64ae-47f7-a556-6799e4146567 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:42.325074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-867f8532-64ae-47f7-a556-6799e4146567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:42.327467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:42.408692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa042a14-9446-426e-8b27-c8d0dd0a7ee9 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:42.412399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 151/298] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:42.412399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e448fa77-3943-4bbc-8e5c-e7df3190f4f2 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:42.412595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e448fa77-3943-4bbc-8e5c-e7df3190f4f2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:42.413488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e448fa77-3943-4bbc-8e5c-e7df3190f4f2 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:42.413954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 152/299] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:42 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:42.419955 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0620ffd-05bd-4f8c-81e2-2c0d372f3aa7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:42.420242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0620ffd-05bd-4f8c-81e2-2c0d372f3aa7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:42.420541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0620ffd-05bd-4f8c-81e2-2c0d372f3aa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:42.439357 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c0620ffd-05bd-4f8c-81e2-2c0d372f3aa7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:42.440673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c0620ffd-05bd-4f8c-81e2-2c0d372f3aa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:42.448001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f801cda-9bf0-4b37-b38c-62f95f017a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:42.448857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f801cda-9bf0-4b37-b38c-62f95f017a0c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:42.449282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f801cda-9bf0-4b37-b38c-62f95f017a0c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:42.449694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f801cda-9bf0-4b37-b38c-62f95f017a0c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:42.460783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f801cda-9bf0-4b37-b38c-62f95f017a0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-1f801cda-9bf0-4b37-b38c-62f95f017a0c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:42.466611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:42.467688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:42.468738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:42.475554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 148/300] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:42.476953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f801cda-9bf0-4b37-b38c-62f95f017a0c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:42.520687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 153/301] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:42.567518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:42.583295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f801cda-9bf0-4b37-b38c-62f95f017a0c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:42.619439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b39ac1f-4331-414e-8407-2c0b54194c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:42.620038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39ac1f-4331-414e-8407-2c0b54194c9e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:42.620319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39ac1f-4331-414e-8407-2c0b54194c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:42.620620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39ac1f-4331-414e-8407-2c0b54194c9e None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:42.621282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b39ac1f-4331-414e-8407-2c0b54194c9e None admin] RBAC: Adding query filter params (name=member) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:42.621584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b39ac1f-4331-414e-8407-2c0b54194c9e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:42.622963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b39ac1f-4331-414e-8407-2c0b54194c9e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:42.636056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 149/302] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:42.637523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0620ffd-05bd-4f8c-81e2-2c0d372f3aa7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:42.697322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:42.726218 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b39ac1f-4331-414e-8407-2c0b54194c9e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:42.749686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0620ffd-05bd-4f8c-81e2-2c0d372f3aa7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:42.796355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f304fb77-dd9a-484b-8678-5e0ccc78f8e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:42.797181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304fb77-dd9a-484b-8678-5e0ccc78f8e7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:42.797717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304fb77-dd9a-484b-8678-5e0ccc78f8e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:42.798109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304fb77-dd9a-484b-8678-5e0ccc78f8e7 None admin] PATH_INFO: `/v3/users/037ab5c772cb4a42a03d3ae398497701` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:42.800139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f304fb77-dd9a-484b-8678-5e0ccc78f8e7 None admin] RBAC: Authorizing `identity:get_user(user_id=037ab5c772cb4a42a03d3ae398497701)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:42.802365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f304fb77-dd9a-484b-8678-5e0ccc78f8e7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:42.804711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 154/303] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:42 2022] GET /identity/v3/users/037ab5c772cb4a42a03d3ae398497701 => generated 293 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:42.809546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f304fb77-dd9a-484b-8678-5e0ccc78f8e7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:42.851973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b39ac1f-4331-414e-8407-2c0b54194c9e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:42.888673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ece020-ac32-4ac5-818a-6bf8972fa97e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:42.889302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ece020-ac32-4ac5-818a-6bf8972fa97e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:42.889578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ece020-ac32-4ac5-818a-6bf8972fa97e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:42.889849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ece020-ac32-4ac5-818a-6bf8972fa97e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:42.901230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ece020-ac32-4ac5-818a-6bf8972fa97e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-02ece020-ac32-4ac5-818a-6bf8972fa97e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:42.906502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:42.907582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:42.908588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:42.912202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 150/304] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:42 2022] GET /identity/v3/roles/reader => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:22:42.913251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ece020-ac32-4ac5-818a-6bf8972fa97e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:42.924147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:42.928833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f304fb77-dd9a-484b-8678-5e0ccc78f8e7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:42.952459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:42.964886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0916e5b1-977c-4601-84ad-e0acb2051606 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:42.965506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0916e5b1-977c-4601-84ad-e0acb2051606 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:42.965783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0916e5b1-977c-4601-84ad-e0acb2051606 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:42.966053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0916e5b1-977c-4601-84ad-e0acb2051606 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:42.967146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0916e5b1-977c-4601-84ad-e0acb2051606 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:42.968615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0916e5b1-977c-4601-84ad-e0acb2051606 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:42.970207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 155/305] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:42.970959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0916e5b1-977c-4601-84ad-e0acb2051606 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:43.017630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ece020-ac32-4ac5-818a-6bf8972fa97e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:43.053299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34bc3c4-7063-445c-9c1c-ce3dcff4efcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:43.053952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34bc3c4-7063-445c-9c1c-ce3dcff4efcb None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:43.054228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34bc3c4-7063-445c-9c1c-ce3dcff4efcb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:43.054499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34bc3c4-7063-445c-9c1c-ce3dcff4efcb None admin] PATH_INFO: `/v3/system/users/037ab5c772cb4a42a03d3ae398497701/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:43.056379 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34bc3c4-7063-445c-9c1c-ce3dcff4efcb None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=037ab5c772cb4a42a03d3ae398497701, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:43.057302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34bc3c4-7063-445c-9c1c-ce3dcff4efcb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:43.065770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 151/306] 10.222.0.25 () {58 vars in 1334 bytes} [Mon Sep 26 01:22:42 2022] PUT /identity/v3/system/users/037ab5c772cb4a42a03d3ae398497701/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:43.074135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a9aa4-3862-4387-8b4c-bd9e1c82de80 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:43.074491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a9aa4-3862-4387-8b4c-bd9e1c82de80 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:43.074694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a9aa4-3862-4387-8b4c-bd9e1c82de80 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:43.076083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 152/307] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:43 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:43.076894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a9aa4-3862-4387-8b4c-bd9e1c82de80 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:43.077442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0916e5b1-977c-4601-84ad-e0acb2051606 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:43.117863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82066881-3736-46d2-841e-578e3b6da58c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:43.118999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82066881-3736-46d2-841e-578e3b6da58c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:43.119294 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82066881-3736-46d2-841e-578e3b6da58c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:43.119557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82066881-3736-46d2-841e-578e3b6da58c None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:43.120256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82066881-3736-46d2-841e-578e3b6da58c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:43.120600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82066881-3736-46d2-841e-578e3b6da58c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:43.122043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82066881-3736-46d2-841e-578e3b6da58c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:43.131574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 156/308] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:42 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:43.132710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f0ccb-b82e-4b2b-9df6-171e887c26d7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:43.133000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f0ccb-b82e-4b2b-9df6-171e887c26d7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:43.133278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f0ccb-b82e-4b2b-9df6-171e887c26d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:43.153067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-622f0ccb-b82e-4b2b-9df6-171e887c26d7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:43.154320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-622f0ccb-b82e-4b2b-9df6-171e887c26d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:43.162966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:43.202391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a9aa4-3862-4387-8b4c-bd9e1c82de80 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:43.216186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:43.244381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 157/309] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:43.244381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622f0ccb-b82e-4b2b-9df6-171e887c26d7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:43.244991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48f9133-728c-4c51-ba97-b1bb470cb61f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:43.245609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48f9133-728c-4c51-ba97-b1bb470cb61f None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:43.245944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48f9133-728c-4c51-ba97-b1bb470cb61f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:43.246218 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48f9133-728c-4c51-ba97-b1bb470cb61f None admin] PATH_INFO: `/v3/groups` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:43.246886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48f9133-728c-4c51-ba97-b1bb470cb61f None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:43.248329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48f9133-728c-4c51-ba97-b1bb470cb61f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:43.267886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 153/310] 10.222.0.25 () {60 vars in 1134 bytes} [Mon Sep 26 01:22:43 2022] POST /identity/v3/groups => generated 229 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:22:43.366361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622f0ccb-b82e-4b2b-9df6-171e887c26d7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:43.407512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02f2685-70ca-4369-a65b-01f30b5c19a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:43.408132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02f2685-70ca-4369-a65b-01f30b5c19a2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:43.408447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02f2685-70ca-4369-a65b-01f30b5c19a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:43.408726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02f2685-70ca-4369-a65b-01f30b5c19a2 None admin] PATH_INFO: `/v3/users/bb3da03725674d9dae97a2ef0970b9a6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:43.424823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02f2685-70ca-4369-a65b-01f30b5c19a2 None admin] RBAC: Authorizing `identity:get_user(user_id=bb3da03725674d9dae97a2ef0970b9a6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:43.426684 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02f2685-70ca-4369-a65b-01f30b5c19a2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:43.428630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 158/311] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:43 2022] GET /identity/v3/users/bb3da03725674d9dae97a2ef0970b9a6 => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:43.435251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48f9133-728c-4c51-ba97-b1bb470cb61f None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:43.519427 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02f2685-70ca-4369-a65b-01f30b5c19a2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:43.561591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48f9133-728c-4c51-ba97-b1bb470cb61f None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:43.573509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:43.599165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:43.604336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aaff4fd-9a20-4f0f-9e09-4aa1ded10c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:43.605171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aaff4fd-9a20-4f0f-9e09-4aa1ded10c40 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:43.605779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aaff4fd-9a20-4f0f-9e09-4aa1ded10c40 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:43.606281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aaff4fd-9a20-4f0f-9e09-4aa1ded10c40 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:43.607110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aaff4fd-9a20-4f0f-9e09-4aa1ded10c40 None admin] RBAC: Adding query filter params (role.id=8922a4a1fc6e4b22815abb10e1f19807, user.id=bb3da03725674d9dae97a2ef0970b9a6, scope.system=all) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:43.607564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aaff4fd-9a20-4f0f-9e09-4aa1ded10c40 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:43.609159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aaff4fd-9a20-4f0f-9e09-4aa1ded10c40 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:43.614944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 154/312] 10.222.0.25 () {56 vars in 1308 bytes} [Mon Sep 26 01:22:43 2022] GET /identity/v3/role_assignments?user.id=bb3da03725674d9dae97a2ef0970b9a6&scope.system=all&role.id=8922a4a1fc6e4b22815abb10e1f19807 => generated 226 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:43.652723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02f2685-70ca-4369-a65b-01f30b5c19a2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:43.694013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0556f7ef-e5c4-423e-b06d-b09d6d93b599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:43.694579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556f7ef-e5c4-423e-b06d-b09d6d93b599 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:43.694859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556f7ef-e5c4-423e-b06d-b09d6d93b599 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:43.695131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556f7ef-e5c4-423e-b06d-b09d6d93b599 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:43.708375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0556f7ef-e5c4-423e-b06d-b09d6d93b599 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0556f7ef-e5c4-423e-b06d-b09d6d93b599 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:43.713916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:43.714865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:43.715652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 159/313] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:43 2022] GET /identity/v3/roles/reader => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:22:43.721152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aaff4fd-9a20-4f0f-9e09-4aa1ded10c40 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:43.838273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aaff4fd-9a20-4f0f-9e09-4aa1ded10c40 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:43.875038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2b22ed-cec0-497c-ab9e-f68f7df5f629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:43.875650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2b22ed-cec0-497c-ab9e-f68f7df5f629 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:43.875927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2b22ed-cec0-497c-ab9e-f68f7df5f629 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:43.876199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2b22ed-cec0-497c-ab9e-f68f7df5f629 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:43.876930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2b22ed-cec0-497c-ab9e-f68f7df5f629 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:43.877234 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2b22ed-cec0-497c-ab9e-f68f7df5f629 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:43.878607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2b22ed-cec0-497c-ab9e-f68f7df5f629 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:43.892746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 155/314] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:43 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:43.894567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0556f7ef-e5c4-423e-b06d-b09d6d93b599 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:44.002843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0556f7ef-e5c4-423e-b06d-b09d6d93b599 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:44.004478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:44.039377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fcaf30-0c9c-4244-b591-1dc7ef5da992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:44.040030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fcaf30-0c9c-4244-b591-1dc7ef5da992 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:44.044392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fcaf30-0c9c-4244-b591-1dc7ef5da992 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:44.044392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fcaf30-0c9c-4244-b591-1dc7ef5da992 None admin] PATH_INFO: `/v3/users/70b8e04c7aa346fd87334f8d292f31f0` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:44.044392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fcaf30-0c9c-4244-b591-1dc7ef5da992 None admin] RBAC: Authorizing `identity:get_user(user_id=70b8e04c7aa346fd87334f8d292f31f0)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:44.044392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fcaf30-0c9c-4244-b591-1dc7ef5da992 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:44.050384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 160/315] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:43 2022] GET /identity/v3/users/70b8e04c7aa346fd87334f8d292f31f0 => generated 297 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:44.053706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2b22ed-cec0-497c-ab9e-f68f7df5f629 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:44.117770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:44.161379 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2b22ed-cec0-497c-ab9e-f68f7df5f629 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:44.195770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b638134-cf8c-49ba-ac85-8b72b031ce33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:44.198202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b638134-cf8c-49ba-ac85-8b72b031ce33 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:44.198627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b638134-cf8c-49ba-ac85-8b72b031ce33 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:44.199116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b638134-cf8c-49ba-ac85-8b72b031ce33 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:44.271540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b638134-cf8c-49ba-ac85-8b72b031ce33 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:44.273238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b638134-cf8c-49ba-ac85-8b72b031ce33 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:44.334450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:44.379243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 156/316] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:44.385077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fcaf30-0c9c-4244-b591-1dc7ef5da992 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:44.492030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fcaf30-0c9c-4244-b591-1dc7ef5da992 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:44.529122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9fffd2-f48b-400e-adee-fd0db77cd072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:44.529678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9fffd2-f48b-400e-adee-fd0db77cd072 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:44.529958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9fffd2-f48b-400e-adee-fd0db77cd072 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:44.530231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9fffd2-f48b-400e-adee-fd0db77cd072 None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:44.531355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9fffd2-f48b-400e-adee-fd0db77cd072 None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:44.532853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9fffd2-f48b-400e-adee-fd0db77cd072 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:44.534784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 161/317] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:44 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:44.540976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b638134-cf8c-49ba-ac85-8b72b031ce33 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:44.655666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b638134-cf8c-49ba-ac85-8b72b031ce33 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:44.693491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8243a0e1-bd9d-4d26-b48a-dbbbf9b563bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:44.694153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243a0e1-bd9d-4d26-b48a-dbbbf9b563bc None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:44.694431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243a0e1-bd9d-4d26-b48a-dbbbf9b563bc None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:44.694702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243a0e1-bd9d-4d26-b48a-dbbbf9b563bc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:44.695597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243a0e1-bd9d-4d26-b48a-dbbbf9b563bc None admin] RBAC: Adding query filter params (role.id=8922a4a1fc6e4b22815abb10e1f19807, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3, user.id=70b8e04c7aa346fd87334f8d292f31f0) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:44.695905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243a0e1-bd9d-4d26-b48a-dbbbf9b563bc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:44.697575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243a0e1-bd9d-4d26-b48a-dbbbf9b563bc None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:44.708479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 157/318] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:22:44 2022] GET /identity/v3/role_assignments?user.id=70b8e04c7aa346fd87334f8d292f31f0&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=8922a4a1fc6e4b22815abb10e1f19807 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:45.384938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424ab6fa-cf7a-4296-89d0-2de7b687dc9c None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:45.385278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424ab6fa-cf7a-4296-89d0-2de7b687dc9c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:45.385793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424ab6fa-cf7a-4296-89d0-2de7b687dc9c None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:45.386908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 162/319] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:22:45.391971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158832f6-bb4b-466d-921f-323ff7cd58b3 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:45.392402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158832f6-bb4b-466d-921f-323ff7cd58b3 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:45.392577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158832f6-bb4b-466d-921f-323ff7cd58b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:45.411423 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-158832f6-bb4b-466d-921f-323ff7cd58b3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:45.412635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-158832f6-bb4b-466d-921f-323ff7cd58b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:45.430012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:45.482244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 158/320] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:45.622351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fff7554-ffc2-4337-92bb-813316d3f84f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:45.622702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fff7554-ffc2-4337-92bb-813316d3f84f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:45.622898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fff7554-ffc2-4337-92bb-813316d3f84f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:45.624403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 163/321] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:45.629827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd8be1-4b2b-44e1-b9e5-c4688c81b77e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:45.630108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd8be1-4b2b-44e1-b9e5-c4688c81b77e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:45.630381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd8be1-4b2b-44e1-b9e5-c4688c81b77e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:45.649443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-6ffd8be1-4b2b-44e1-b9e5-c4688c81b77e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:45.650753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6ffd8be1-4b2b-44e1-b9e5-c4688c81b77e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:45.724389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 159/322] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:45.750154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fff7554-ffc2-4337-92bb-813316d3f84f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:45.793886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:45.855918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fff7554-ffc2-4337-92bb-813316d3f84f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:45.868066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db0e592-5c2c-4699-9200-69fa153e3be1 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:45.868420 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db0e592-5c2c-4699-9200-69fa153e3be1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:45.868811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db0e592-5c2c-4699-9200-69fa153e3be1 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:45.870614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 160/323] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:45.875195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79170fcd-260d-49f4-86cf-b12d8da6e21e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:45.875477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79170fcd-260d-49f4-86cf-b12d8da6e21e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:45.875744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79170fcd-260d-49f4-86cf-b12d8da6e21e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:45.890659 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7d32f5-2d10-4cc5-8543-6ae268f68491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:45.891280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d32f5-2d10-4cc5-8543-6ae268f68491 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:45.891553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d32f5-2d10-4cc5-8543-6ae268f68491 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:45.891821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d32f5-2d10-4cc5-8543-6ae268f68491 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:45.894814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-79170fcd-260d-49f4-86cf-b12d8da6e21e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:45.896064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-79170fcd-260d-49f4-86cf-b12d8da6e21e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:45.903069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7d32f5-2d10-4cc5-8543-6ae268f68491 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-da7d32f5-2d10-4cc5-8543-6ae268f68491 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:45.907822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:45.908849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:45.909645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 164/324] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:45.914541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7d32f5-2d10-4cc5-8543-6ae268f68491 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:45.979364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 161/325] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:45.989139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79170fcd-260d-49f4-86cf-b12d8da6e21e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:46.004687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:46.018061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:46.021781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7d32f5-2d10-4cc5-8543-6ae268f68491 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:46.066324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4212c-83b4-4a4c-bce0-da8e998f21e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:46.066928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4212c-83b4-4a4c-bce0-da8e998f21e8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:46.067209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4212c-83b4-4a4c-bce0-da8e998f21e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:46.067480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4212c-83b4-4a4c-bce0-da8e998f21e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:46.068140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a4212c-83b4-4a4c-bce0-da8e998f21e8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:46.068634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a4212c-83b4-4a4c-bce0-da8e998f21e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:46.070141 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a4212c-83b4-4a4c-bce0-da8e998f21e8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:46.079400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 165/326] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:46.085023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4212c-83b4-4a4c-bce0-da8e998f21e8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:46.099223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79170fcd-260d-49f4-86cf-b12d8da6e21e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:46.136395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dd08b2-4e23-44c5-9be9-d9cd2dd0fca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:46.137025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dd08b2-4e23-44c5-9be9-d9cd2dd0fca2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:46.137303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dd08b2-4e23-44c5-9be9-d9cd2dd0fca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:46.137573 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dd08b2-4e23-44c5-9be9-d9cd2dd0fca2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:46.138795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dd08b2-4e23-44c5-9be9-d9cd2dd0fca2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:46.140227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dd08b2-4e23-44c5-9be9-d9cd2dd0fca2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:46.142325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 162/327] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:22:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:46.147494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dd08b2-4e23-44c5-9be9-d9cd2dd0fca2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:46.189215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4212c-83b4-4a4c-bce0-da8e998f21e8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:46.223045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1daf5a-5599-4aa5-af82-da40e916b0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:46.223703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1daf5a-5599-4aa5-af82-da40e916b0c8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:46.223962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1daf5a-5599-4aa5-af82-da40e916b0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:46.224237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1daf5a-5599-4aa5-af82-da40e916b0c8 None admin] PATH_INFO: `/v3/users/037ab5c772cb4a42a03d3ae398497701` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:46.225387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1daf5a-5599-4aa5-af82-da40e916b0c8 None admin] RBAC: Authorizing `identity:get_user(user_id=037ab5c772cb4a42a03d3ae398497701)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:46.226831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1daf5a-5599-4aa5-af82-da40e916b0c8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:46.229092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 166/328] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:46 2022] GET /identity/v3/users/037ab5c772cb4a42a03d3ae398497701 => generated 293 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:46.234834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1daf5a-5599-4aa5-af82-da40e916b0c8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:46.238560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:46.260223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dd08b2-4e23-44c5-9be9-d9cd2dd0fca2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:46.262817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:46.295467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fae8d3f-983b-4bfd-a27f-05d28650baae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:46.296069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fae8d3f-983b-4bfd-a27f-05d28650baae None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:46.296368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fae8d3f-983b-4bfd-a27f-05d28650baae None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:46.296653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fae8d3f-983b-4bfd-a27f-05d28650baae None admin] PATH_INFO: `/v3/groups` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:46.297283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fae8d3f-983b-4bfd-a27f-05d28650baae None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:46.298740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fae8d3f-983b-4bfd-a27f-05d28650baae None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:46.313788 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 163/329] 10.222.0.25 () {60 vars in 1134 bytes} [Mon Sep 26 01:22:46 2022] POST /identity/v3/groups => generated 226 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:22:46.314788 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fae8d3f-983b-4bfd-a27f-05d28650baae None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:46.361884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1daf5a-5599-4aa5-af82-da40e916b0c8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:46.409763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82273ed1-7174-4131-9d25-a1382dba4e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:46.410409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82273ed1-7174-4131-9d25-a1382dba4e47 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:46.410643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82273ed1-7174-4131-9d25-a1382dba4e47 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:46.410911 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82273ed1-7174-4131-9d25-a1382dba4e47 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:46.412156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82273ed1-7174-4131-9d25-a1382dba4e47 None admin] RBAC: Adding query filter params (role.id=27476172dbb84c6e86b9070e02d06100, user.id=037ab5c772cb4a42a03d3ae398497701, scope.system=all) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:46.412506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82273ed1-7174-4131-9d25-a1382dba4e47 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:46.413994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82273ed1-7174-4131-9d25-a1382dba4e47 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:46.427066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fae8d3f-983b-4bfd-a27f-05d28650baae None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:46.427787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 167/330] 10.222.0.25 () {56 vars in 1308 bytes} [Mon Sep 26 01:22:46 2022] GET /identity/v3/role_assignments?user.id=037ab5c772cb4a42a03d3ae398497701&scope.system=all&role.id=27476172dbb84c6e86b9070e02d06100 => generated 510 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:46.472807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a30151-2f84-451c-8d24-ab937ddbdc48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:46.473467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a30151-2f84-451c-8d24-ab937ddbdc48 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:46.473746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a30151-2f84-451c-8d24-ab937ddbdc48 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:46.474019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a30151-2f84-451c-8d24-ab937ddbdc48 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:46.483856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:46.487885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a30151-2f84-451c-8d24-ab937ddbdc48 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-81a30151-2f84-451c-8d24-ab937ddbdc48 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:46.494445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:46.495589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:46.496445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:46.499056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 164/331] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:46 2022] GET /identity/v3/roles/reader => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:46.505559 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82273ed1-7174-4131-9d25-a1382dba4e47 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:46.605610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:46.622873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82273ed1-7174-4131-9d25-a1382dba4e47 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:46.660584 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d5e7f5-e17f-4d5e-8b56-849de705a74f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:46.661236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d5e7f5-e17f-4d5e-8b56-849de705a74f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:46.661517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d5e7f5-e17f-4d5e-8b56-849de705a74f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:46.661790 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d5e7f5-e17f-4d5e-8b56-849de705a74f None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:46.663040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d5e7f5-e17f-4d5e-8b56-849de705a74f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:46.663040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d5e7f5-e17f-4d5e-8b56-849de705a74f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:46.664251 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d5e7f5-e17f-4d5e-8b56-849de705a74f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:46.674074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 168/332] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:46 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:46.679878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a30151-2f84-451c-8d24-ab937ddbdc48 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:46.796015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a30151-2f84-451c-8d24-ab937ddbdc48 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:46.833147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d6ccb8-4d5c-4abb-a39d-9239926ee13e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:46.833794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d6ccb8-4d5c-4abb-a39d-9239926ee13e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:46.834073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d6ccb8-4d5c-4abb-a39d-9239926ee13e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:46.834341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d6ccb8-4d5c-4abb-a39d-9239926ee13e None admin] PATH_INFO: `/v3/users/bb3da03725674d9dae97a2ef0970b9a6` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:46.835492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d6ccb8-4d5c-4abb-a39d-9239926ee13e None admin] RBAC: Authorizing `identity:get_user(user_id=bb3da03725674d9dae97a2ef0970b9a6)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:46.836972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d6ccb8-4d5c-4abb-a39d-9239926ee13e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:46.839435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 165/333] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:46 2022] GET /identity/v3/users/bb3da03725674d9dae97a2ef0970b9a6 => generated 293 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:46.845500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d5e7f5-e17f-4d5e-8b56-849de705a74f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:46.968531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d5e7f5-e17f-4d5e-8b56-849de705a74f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:47.006963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939bb82a-f129-4a2c-a6df-1baf370e9bbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:47.007976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939bb82a-f129-4a2c-a6df-1baf370e9bbf None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:47.008483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939bb82a-f129-4a2c-a6df-1baf370e9bbf None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:47.009015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939bb82a-f129-4a2c-a6df-1baf370e9bbf None admin] PATH_INFO: `/v3/system/users/bb3da03725674d9dae97a2ef0970b9a6/roles/8922a4a1fc6e4b22815abb10e1f19807` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:47.011204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939bb82a-f129-4a2c-a6df-1baf370e9bbf None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=bb3da03725674d9dae97a2ef0970b9a6, role_id=8922a4a1fc6e4b22815abb10e1f19807)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:47.012873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939bb82a-f129-4a2c-a6df-1baf370e9bbf None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:47.024189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 169/334] 10.222.0.25 () {58 vars in 1334 bytes} [Mon Sep 26 01:22:46 2022] PUT /identity/v3/system/users/bb3da03725674d9dae97a2ef0970b9a6/roles/8922a4a1fc6e4b22815abb10e1f19807 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:48.872858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce544b5-1293-4113-8266-78fc1eeb9d8c None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:48.873366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce544b5-1293-4113-8266-78fc1eeb9d8c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:48.873479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce544b5-1293-4113-8266-78fc1eeb9d8c None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:48.875008 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 166/335] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:48.886438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273285f7-4c7c-45fb-88a2-da1022ac1cce None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:48.886894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273285f7-4c7c-45fb-88a2-da1022ac1cce None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:48.886967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273285f7-4c7c-45fb-88a2-da1022ac1cce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:48.909759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-273285f7-4c7c-45fb-88a2-da1022ac1cce None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:48.911009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-273285f7-4c7c-45fb-88a2-da1022ac1cce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:48.922567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:48.994984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 170/336] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:49.147471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb2e5e8-40c4-4a77-a6f1-de4aa5026db1 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:49.147659 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb2e5e8-40c4-4a77-a6f1-de4aa5026db1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:49.147919 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb2e5e8-40c4-4a77-a6f1-de4aa5026db1 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:49.149844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 167/337] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:49.156239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fdc659-2e29-40c2-9900-c57fa8d7f63a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:49.156546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fdc659-2e29-40c2-9900-c57fa8d7f63a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:49.157080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fdc659-2e29-40c2-9900-c57fa8d7f63a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:49.181762 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d8fdc659-2e29-40c2-9900-c57fa8d7f63a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:49.183300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d8fdc659-2e29-40c2-9900-c57fa8d7f63a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:49.260917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 171/338] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:49.455393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb2e5e8-40c4-4a77-a6f1-de4aa5026db1 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:49.539769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:49.583654 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb2e5e8-40c4-4a77-a6f1-de4aa5026db1 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:49.624825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ad1b6-4a1c-4f10-be17-876c4e2e7006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:49.625460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ad1b6-4a1c-4f10-be17-876c4e2e7006 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:49.625743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ad1b6-4a1c-4f10-be17-876c4e2e7006 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:49.626013 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ad1b6-4a1c-4f10-be17-876c4e2e7006 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:49.641491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4ad1b6-4a1c-4f10-be17-876c4e2e7006 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ea4ad1b6-4a1c-4f10-be17-876c4e2e7006 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:49.646389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:49.647291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:49.648084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:49.649204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 168/339] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:49.654389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fdc659-2e29-40c2-9900-c57fa8d7f63a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:49.763078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:49.795036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fdc659-2e29-40c2-9900-c57fa8d7f63a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:49.843034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d10e021-8354-4641-8395-988c8ea9b707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:49.844014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d10e021-8354-4641-8395-988c8ea9b707 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:49.844520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d10e021-8354-4641-8395-988c8ea9b707 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:49.845013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d10e021-8354-4641-8395-988c8ea9b707 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:49.845889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d10e021-8354-4641-8395-988c8ea9b707 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:49.846389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d10e021-8354-4641-8395-988c8ea9b707 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:49.847965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d10e021-8354-4641-8395-988c8ea9b707 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:49.860680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 172/340] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:49.867317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ad1b6-4a1c-4f10-be17-876c4e2e7006 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:49.900281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d10e021-8354-4641-8395-988c8ea9b707 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:50.010102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ad1b6-4a1c-4f10-be17-876c4e2e7006 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:50.052773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:50.069967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fa8fae-079b-4ff4-9e1f-6174d03a5f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:50.070588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fa8fae-079b-4ff4-9e1f-6174d03a5f6e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.070870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fa8fae-079b-4ff4-9e1f-6174d03a5f6e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.071145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fa8fae-079b-4ff4-9e1f-6174d03a5f6e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.083567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d10e021-8354-4641-8395-988c8ea9b707 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:50.129978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612240ab-f82e-4f80-89a3-b29ef950c4a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:50.130973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612240ab-f82e-4f80-89a3-b29ef950c4a0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.131451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612240ab-f82e-4f80-89a3-b29ef950c4a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.131925 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612240ab-f82e-4f80-89a3-b29ef950c4a0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.140598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:50.155167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612240ab-f82e-4f80-89a3-b29ef950c4a0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-612240ab-f82e-4f80-89a3-b29ef950c4a0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:50.164756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:50.166299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:50.167096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:50.169468 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 173/341] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:49 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:22:50.172977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fa8fae-079b-4ff4-9e1f-6174d03a5f6e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:50.174913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fa8fae-079b-4ff4-9e1f-6174d03a5f6e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:50.175528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612240ab-f82e-4f80-89a3-b29ef950c4a0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:50.292500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 169/342] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:50.295059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fa8fae-079b-4ff4-9e1f-6174d03a5f6e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:50.348872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:50.370851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612240ab-f82e-4f80-89a3-b29ef950c4a0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:50.410296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fa8fae-079b-4ff4-9e1f-6174d03a5f6e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:50.449938 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43a9dcb-4ffc-4675-aea1-a3ba438e2578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:50.450579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43a9dcb-4ffc-4675-aea1-a3ba438e2578 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.450853 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43a9dcb-4ffc-4675-aea1-a3ba438e2578 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.451121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43a9dcb-4ffc-4675-aea1-a3ba438e2578 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.452228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c99084-3045-4196-990d-73d752d73513 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:50.452858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c99084-3045-4196-990d-73d752d73513 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.453131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c99084-3045-4196-990d-73d752d73513 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.453393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c99084-3045-4196-990d-73d752d73513 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.454056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c99084-3045-4196-990d-73d752d73513 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:50.454351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c99084-3045-4196-990d-73d752d73513 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:50.455988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c99084-3045-4196-990d-73d752d73513 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:50.457012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43a9dcb-4ffc-4675-aea1-a3ba438e2578 None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:50.458434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43a9dcb-4ffc-4675-aea1-a3ba438e2578 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:50.460097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 170/343] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:50 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:50.472677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43a9dcb-4ffc-4675-aea1-a3ba438e2578 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:50.485563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 174/344] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:50.494241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c99084-3045-4196-990d-73d752d73513 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:50.599688 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:50.616748 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:50.622023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43a9dcb-4ffc-4675-aea1-a3ba438e2578 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:50.664830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c99084-3045-4196-990d-73d752d73513 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:50.666420 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc87f02f-0bde-448a-9252-7991eee326b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:50.667209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87f02f-0bde-448a-9252-7991eee326b7 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.667566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87f02f-0bde-448a-9252-7991eee326b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.667910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87f02f-0bde-448a-9252-7991eee326b7 None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.675057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc87f02f-0bde-448a-9252-7991eee326b7 None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:50.676576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc87f02f-0bde-448a-9252-7991eee326b7 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:50.679303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 171/345] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:22:50 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:50.680653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a20a4f0-112e-46f3-9ffb-87d8a2e5ffb4 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.681472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a20a4f0-112e-46f3-9ffb-87d8a2e5ffb4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.681832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a20a4f0-112e-46f3-9ffb-87d8a2e5ffb4 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.683794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 172/346] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:50.685312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e1caec-bf10-458a-b1ea-996d7f92b4bc None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.685596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e1caec-bf10-458a-b1ea-996d7f92b4bc None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.685868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e1caec-bf10-458a-b1ea-996d7f92b4bc None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.687390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 173/347] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:50.688223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e1caec-bf10-458a-b1ea-996d7f92b4bc None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:50.711818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c21eaed-b50b-484d-8911-cd52643c819e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:50.712567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c21eaed-b50b-484d-8911-cd52643c819e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.712848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c21eaed-b50b-484d-8911-cd52643c819e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.713120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c21eaed-b50b-484d-8911-cd52643c819e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.800831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c21eaed-b50b-484d-8911-cd52643c819e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:50.802495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c21eaed-b50b-484d-8911-cd52643c819e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:50.810781 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e1caec-bf10-458a-b1ea-996d7f92b4bc None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:50.853276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99f9799-baea-4b83-8283-9a3c34a570ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:50.854154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99f9799-baea-4b83-8283-9a3c34a570ed None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.854493 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99f9799-baea-4b83-8283-9a3c34a570ed None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.854765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99f9799-baea-4b83-8283-9a3c34a570ed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.856972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99f9799-baea-4b83-8283-9a3c34a570ed None admin] RBAC: Adding query filter params (group.id=41b57b9e63f14cb3afba7976a393b32a, role.id=27476172dbb84c6e86b9070e02d06100, scope.project.id=5f976380e1494db480dc80868d51a0bc) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:50.857892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99f9799-baea-4b83-8283-9a3c34a570ed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:50.859657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99f9799-baea-4b83-8283-9a3c34a570ed None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:50.869031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 174/348] 10.222.0.25 () {56 vars in 1376 bytes} [Mon Sep 26 01:22:50 2022] GET /identity/v3/role_assignments?group.id=41b57b9e63f14cb3afba7976a393b32a&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=27476172dbb84c6e86b9070e02d06100 => generated 260 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:50.870239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e5fbbe-3969-4118-a2ef-6dbb27047587 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.870915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e5fbbe-3969-4118-a2ef-6dbb27047587 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.870915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e5fbbe-3969-4118-a2ef-6dbb27047587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.877816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:50.895082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-80e5fbbe-3969-4118-a2ef-6dbb27047587 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:50.896313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-80e5fbbe-3969-4118-a2ef-6dbb27047587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:50.949255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 175/349] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 456 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:50.951817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5584ea4-1501-4d2c-b005-84f81460fc46 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:50.952311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5584ea4-1501-4d2c-b005-84f81460fc46 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:50.953249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5584ea4-1501-4d2c-b005-84f81460fc46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:50.964415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:50.999239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e5584ea4-1501-4d2c-b005-84f81460fc46 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:51.000572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e5584ea4-1501-4d2c-b005-84f81460fc46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:51.001485 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 175/350] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:51.003031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234e0df3-e387-44e3-bdcb-8da71c21488a None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:51.003497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234e0df3-e387-44e3-bdcb-8da71c21488a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:51.003974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234e0df3-e387-44e3-bdcb-8da71c21488a None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:51.005547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 176/351] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:51.006474 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234e0df3-e387-44e3-bdcb-8da71c21488a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:51.144300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 176/352] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:51.145376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c44d39-6af5-48cc-862a-5f65711f3f8b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:51.145672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c44d39-6af5-48cc-862a-5f65711f3f8b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:51.145929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c44d39-6af5-48cc-862a-5f65711f3f8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:51.166940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d5c44d39-6af5-48cc-862a-5f65711f3f8b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:51.168578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d5c44d39-6af5-48cc-862a-5f65711f3f8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:51.169579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234e0df3-e387-44e3-bdcb-8da71c21488a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:51.211919 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8948c760-8f4c-4b38-b89f-7e0d5c122fea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:51.213137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8948c760-8f4c-4b38-b89f-7e0d5c122fea None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:51.213467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8948c760-8f4c-4b38-b89f-7e0d5c122fea None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:51.213813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8948c760-8f4c-4b38-b89f-7e0d5c122fea None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:51.215025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8948c760-8f4c-4b38-b89f-7e0d5c122fea None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:51.216561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8948c760-8f4c-4b38-b89f-7e0d5c122fea None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:51.218802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 177/353] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:51 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:51.220110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ae4a3e-9b18-4aad-817a-db53c236a022 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:51.220404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ae4a3e-9b18-4aad-817a-db53c236a022 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:51.220678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ae4a3e-9b18-4aad-817a-db53c236a022 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:51.226970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 178/354] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:51.228500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ae4a3e-9b18-4aad-817a-db53c236a022 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:51.261739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:51.309658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 177/355] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:51.310927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9b84f6-1abe-417f-ab53-aab426de09fa None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:51.311215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9b84f6-1abe-417f-ab53-aab426de09fa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:51.311487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9b84f6-1abe-417f-ab53-aab426de09fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:51.341667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-cf9b84f6-1abe-417f-ab53-aab426de09fa None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:51.342687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cf9b84f6-1abe-417f-ab53-aab426de09fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:51.357665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ae4a3e-9b18-4aad-817a-db53c236a022 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:51.357839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:51.426115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a36f499-b44d-4004-b69e-9a80050f1c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:51.426891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36f499-b44d-4004-b69e-9a80050f1c9c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:51.427172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36f499-b44d-4004-b69e-9a80050f1c9c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:51.427441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36f499-b44d-4004-b69e-9a80050f1c9c None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:51.430185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a36f499-b44d-4004-b69e-9a80050f1c9c None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:51.434103 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 178/356] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:51.434908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a36f499-b44d-4004-b69e-9a80050f1c9c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:51.435532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9b84f6-1abe-417f-ab53-aab426de09fa None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:51.438716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 179/357] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:22:51 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:51.444116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a36f499-b44d-4004-b69e-9a80050f1c9c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:51.592496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9b84f6-1abe-417f-ab53-aab426de09fa None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:51.619807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:51.629507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae98d734-5db9-4d64-8663-baae4a4dadd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:51.630150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae98d734-5db9-4d64-8663-baae4a4dadd8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:51.630430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae98d734-5db9-4d64-8663-baae4a4dadd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:51.630704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae98d734-5db9-4d64-8663-baae4a4dadd8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:51.643510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae98d734-5db9-4d64-8663-baae4a4dadd8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:51.644152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a36f499-b44d-4004-b69e-9a80050f1c9c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ae98d734-5db9-4d64-8663-baae4a4dadd8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:51.649575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:51.650541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:51.651395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 179/358] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:51.652589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae98d734-5db9-4d64-8663-baae4a4dadd8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:51.666792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:51.686999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dd70d3-a080-4ce7-bc37-8cdf9d4870bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:51.687563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dd70d3-a080-4ce7-bc37-8cdf9d4870bd None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:51.687845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dd70d3-a080-4ce7-bc37-8cdf9d4870bd None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:51.688117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dd70d3-a080-4ce7-bc37-8cdf9d4870bd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:51.688885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dd70d3-a080-4ce7-bc37-8cdf9d4870bd None admin] RBAC: Adding query filter params (group.id=41b57b9e63f14cb3afba7976a393b32a, role.id=48389faedba2470a885fae19084bb456, scope.project.id=5f976380e1494db480dc80868d51a0bc) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:51.689194 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dd70d3-a080-4ce7-bc37-8cdf9d4870bd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:51.691679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dd70d3-a080-4ce7-bc37-8cdf9d4870bd None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:51.699913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 180/359] 10.222.0.25 () {56 vars in 1376 bytes} [Mon Sep 26 01:22:51 2022] GET /identity/v3/role_assignments?group.id=41b57b9e63f14cb3afba7976a393b32a&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=48389faedba2470a885fae19084bb456 => generated 260 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:51.700822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dd70d3-a080-4ce7-bc37-8cdf9d4870bd None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:51.825748 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dd70d3-a080-4ce7-bc37-8cdf9d4870bd None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:51.867286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6476438-30ec-4c47-8e87-a2c7cf68f5e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:51.867933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6476438-30ec-4c47-8e87-a2c7cf68f5e8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:51.868214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6476438-30ec-4c47-8e87-a2c7cf68f5e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:51.868837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6476438-30ec-4c47-8e87-a2c7cf68f5e8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:51.883046 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6476438-30ec-4c47-8e87-a2c7cf68f5e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-b6476438-30ec-4c47-8e87-a2c7cf68f5e8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:51.889375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:51.890345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:51.891152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 181/360] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:51 2022] GET /identity/v3/roles/reader => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:51.891827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6476438-30ec-4c47-8e87-a2c7cf68f5e8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:51.901575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae98d734-5db9-4d64-8663-baae4a4dadd8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:51.918995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:51.955514 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff8dfd0-a0c3-4ea9-a9bf-77e94ceb9ab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:51.959002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff8dfd0-a0c3-4ea9-a9bf-77e94ceb9ab6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:51.959002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff8dfd0-a0c3-4ea9-a9bf-77e94ceb9ab6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:51.959002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff8dfd0-a0c3-4ea9-a9bf-77e94ceb9ab6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:51.976845 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff8dfd0-a0c3-4ea9-a9bf-77e94ceb9ab6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-2ff8dfd0-a0c3-4ea9-a9bf-77e94ceb9ab6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:51.982449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:51.983354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:51.984173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:51.984940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 180/361] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:51 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:51.985481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff8dfd0-a0c3-4ea9-a9bf-77e94ceb9ab6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:52.029341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6476438-30ec-4c47-8e87-a2c7cf68f5e8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:52.062571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:52.083189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864bcdf8-def4-411a-93e0-7a241fd4d796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:52.083815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864bcdf8-def4-411a-93e0-7a241fd4d796 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:52.084099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864bcdf8-def4-411a-93e0-7a241fd4d796 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:52.084408 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864bcdf8-def4-411a-93e0-7a241fd4d796 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:52.085077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864bcdf8-def4-411a-93e0-7a241fd4d796 None admin] RBAC: Adding query filter params (name=member) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:52.085382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864bcdf8-def4-411a-93e0-7a241fd4d796 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:52.086774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864bcdf8-def4-411a-93e0-7a241fd4d796 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:52.095843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 182/362] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:52.097105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864bcdf8-def4-411a-93e0-7a241fd4d796 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:52.113184 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff8dfd0-a0c3-4ea9-a9bf-77e94ceb9ab6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:52.154167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7d674a-ce2e-41b5-8b8a-a3d38876a850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:52.154825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d674a-ce2e-41b5-8b8a-a3d38876a850 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:52.155109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d674a-ce2e-41b5-8b8a-a3d38876a850 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:52.155386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d674a-ce2e-41b5-8b8a-a3d38876a850 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:52.167685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7d674a-ce2e-41b5-8b8a-a3d38876a850 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-3f7d674a-ce2e-41b5-8b8a-a3d38876a850 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:52.174458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:52.175346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:52.176149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 181/363] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:22:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:52.177410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7d674a-ce2e-41b5-8b8a-a3d38876a850 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:52.192069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:52.217508 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864bcdf8-def4-411a-93e0-7a241fd4d796 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:52.255029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a1cfbc-76fd-4155-986d-02b8c6523a23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:52.255812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a1cfbc-76fd-4155-986d-02b8c6523a23 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:52.256250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a1cfbc-76fd-4155-986d-02b8c6523a23 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:52.256663 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a1cfbc-76fd-4155-986d-02b8c6523a23 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:52.257693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a1cfbc-76fd-4155-986d-02b8c6523a23 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:52.257949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a1cfbc-76fd-4155-986d-02b8c6523a23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:52.259972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a1cfbc-76fd-4155-986d-02b8c6523a23 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:52.268802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 183/364] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:52.269694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a1cfbc-76fd-4155-986d-02b8c6523a23 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:52.313410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7d674a-ce2e-41b5-8b8a-a3d38876a850 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:52.347546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:52.359900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f987f296-58b2-494f-8bff-f8225541a674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:52.360841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f987f296-58b2-494f-8bff-f8225541a674 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:52.361302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f987f296-58b2-494f-8bff-f8225541a674 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:52.361778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f987f296-58b2-494f-8bff-f8225541a674 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:52.362643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f987f296-58b2-494f-8bff-f8225541a674 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:52.363124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f987f296-58b2-494f-8bff-f8225541a674 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:52.365214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f987f296-58b2-494f-8bff-f8225541a674 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:52.379392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 182/365] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:52.381743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f987f296-58b2-494f-8bff-f8225541a674 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:52.404083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a1cfbc-76fd-4155-986d-02b8c6523a23 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:52.449275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862c6f67-c320-43fe-843b-be936fdf10f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:52.450230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862c6f67-c320-43fe-843b-be936fdf10f5 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:52.450520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862c6f67-c320-43fe-843b-be936fdf10f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:52.451061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862c6f67-c320-43fe-843b-be936fdf10f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:52.462946 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:52.512099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f987f296-58b2-494f-8bff-f8225541a674 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:52.538811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862c6f67-c320-43fe-843b-be936fdf10f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:52.541868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862c6f67-c320-43fe-843b-be936fdf10f5 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:52.555063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c01351-8f32-4f76-be8a-2dfe7b02e8d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:52.556180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c01351-8f32-4f76-be8a-2dfe7b02e8d8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:52.556522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c01351-8f32-4f76-be8a-2dfe7b02e8d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:52.557088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c01351-8f32-4f76-be8a-2dfe7b02e8d8 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:52.557774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c01351-8f32-4f76-be8a-2dfe7b02e8d8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:52.558086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c01351-8f32-4f76-be8a-2dfe7b02e8d8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:52.560078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c01351-8f32-4f76-be8a-2dfe7b02e8d8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:52.571193 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 183/366] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:52.572625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c01351-8f32-4f76-be8a-2dfe7b02e8d8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:52.638642 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:52.672329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 184/367] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:52.678609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862c6f67-c320-43fe-843b-be936fdf10f5 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:52.705471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c01351-8f32-4f76-be8a-2dfe7b02e8d8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:52.729320 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:52.753486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f1d85d-fe6d-4c35-a732-de3e2dc2d476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:52.754103 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f1d85d-fe6d-4c35-a732-de3e2dc2d476 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:52.754379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f1d85d-fe6d-4c35-a732-de3e2dc2d476 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:52.754647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f1d85d-fe6d-4c35-a732-de3e2dc2d476 None admin] PATH_INFO: `/v3/users/bb3da03725674d9dae97a2ef0970b9a6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:52.755899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f1d85d-fe6d-4c35-a732-de3e2dc2d476 None admin] RBAC: Authorizing `identity:get_user(user_id=bb3da03725674d9dae97a2ef0970b9a6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:52.757592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f1d85d-fe6d-4c35-a732-de3e2dc2d476 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:52.759959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 184/368] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:22:52 2022] GET /identity/v3/users/bb3da03725674d9dae97a2ef0970b9a6 => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:52.760974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f1d85d-fe6d-4c35-a732-de3e2dc2d476 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:52.807702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862c6f67-c320-43fe-843b-be936fdf10f5 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:52.852017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635c69ca-4a1e-4ec5-9a8c-a1ec5516e6cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:52.852384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635c69ca-4a1e-4ec5-9a8c-a1ec5516e6cb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:52.855159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635c69ca-4a1e-4ec5-9a8c-a1ec5516e6cb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:52.855437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635c69ca-4a1e-4ec5-9a8c-a1ec5516e6cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:52.896979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f1d85d-fe6d-4c35-a732-de3e2dc2d476 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:52.919874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:52.935909 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635c69ca-4a1e-4ec5-9a8c-a1ec5516e6cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:52.937567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635c69ca-4a1e-4ec5-9a8c-a1ec5516e6cb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:52.943229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb70ae98-3186-4ab0-b860-c69c1c096840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:52.944043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb70ae98-3186-4ab0-b860-c69c1c096840 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:52.944510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb70ae98-3186-4ab0-b860-c69c1c096840 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:52.944946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb70ae98-3186-4ab0-b860-c69c1c096840 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:52.989869 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:53.029511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb70ae98-3186-4ab0-b860-c69c1c096840 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:53.031970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb70ae98-3186-4ab0-b860-c69c1c096840 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:53.079383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 185/369] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:53.084839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635c69ca-4a1e-4ec5-9a8c-a1ec5516e6cb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:53.158898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 185/370] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:53.159845 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb70ae98-3186-4ab0-b860-c69c1c096840 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:53.180128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:53.214364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635c69ca-4a1e-4ec5-9a8c-a1ec5516e6cb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:53.257117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbe5653-d728-4a1f-80e7-141898519b66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:53.257747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbe5653-d728-4a1f-80e7-141898519b66 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:53.258022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbe5653-d728-4a1f-80e7-141898519b66 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:53.258292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbe5653-d728-4a1f-80e7-141898519b66 None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:53.259617 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbe5653-d728-4a1f-80e7-141898519b66 None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:53.261608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbe5653-d728-4a1f-80e7-141898519b66 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:53.264087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 186/371] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:53 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:53.265184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbe5653-d728-4a1f-80e7-141898519b66 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:53.276863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:53.289473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb70ae98-3186-4ab0-b860-c69c1c096840 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:53.343841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce92a9-e77b-4e14-b806-7128ef2b5742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:53.344487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce92a9-e77b-4e14-b806-7128ef2b5742 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:53.344830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce92a9-e77b-4e14-b806-7128ef2b5742 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:53.345166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce92a9-e77b-4e14-b806-7128ef2b5742 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:53.346527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce92a9-e77b-4e14-b806-7128ef2b5742 None admin] RBAC: Adding query filter params (role.id=8922a4a1fc6e4b22815abb10e1f19807, user.id=bb3da03725674d9dae97a2ef0970b9a6, scope.system=all) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:53.346908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce92a9-e77b-4e14-b806-7128ef2b5742 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:53.349195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce92a9-e77b-4e14-b806-7128ef2b5742 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:53.372137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 186/372] 10.222.0.25 () {56 vars in 1308 bytes} [Mon Sep 26 01:22:53 2022] GET /identity/v3/role_assignments?user.id=bb3da03725674d9dae97a2ef0970b9a6&scope.system=all&role.id=8922a4a1fc6e4b22815abb10e1f19807 => generated 510 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:53.372137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce92a9-e77b-4e14-b806-7128ef2b5742 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:53.423877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbe5653-d728-4a1f-80e7-141898519b66 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:53.475578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:53.478375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43c07b3-2152-4e38-a45e-582e9c31eb8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:53.479606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43c07b3-2152-4e38-a45e-582e9c31eb8d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:53.480079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43c07b3-2152-4e38-a45e-582e9c31eb8d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:53.480587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43c07b3-2152-4e38-a45e-582e9c31eb8d None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:53.484898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43c07b3-2152-4e38-a45e-582e9c31eb8d None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:53.486720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43c07b3-2152-4e38-a45e-582e9c31eb8d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:53.492483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 187/373] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:53 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:53.494257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43c07b3-2152-4e38-a45e-582e9c31eb8d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:53.514700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce92a9-e77b-4e14-b806-7128ef2b5742 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:53.558809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47137db3-9e08-4e90-8ae3-5cf3f3f0af63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:53.559550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47137db3-9e08-4e90-8ae3-5cf3f3f0af63 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:53.559897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47137db3-9e08-4e90-8ae3-5cf3f3f0af63 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:53.560236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47137db3-9e08-4e90-8ae3-5cf3f3f0af63 None admin] PATH_INFO: `/v3/projects/7124d36dd410484faf5f0034e732bac5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:53.561884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47137db3-9e08-4e90-8ae3-5cf3f3f0af63 None admin] RBAC: Authorizing `identity:get_project(project_id=7124d36dd410484faf5f0034e732bac5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:53.564123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47137db3-9e08-4e90-8ae3-5cf3f3f0af63 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:53.566035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 187/374] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:53 2022] GET /identity/v3/projects/7124d36dd410484faf5f0034e732bac5 => generated 344 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:53.566921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47137db3-9e08-4e90-8ae3-5cf3f3f0af63 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:53.633236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:53.670171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43c07b3-2152-4e38-a45e-582e9c31eb8d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:53.693064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47137db3-9e08-4e90-8ae3-5cf3f3f0af63 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:53.712590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc1602c-11d8-40b7-9227-6b1263aa006d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:53.713252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc1602c-11d8-40b7-9227-6b1263aa006d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:53.713969 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc1602c-11d8-40b7-9227-6b1263aa006d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:53.714245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc1602c-11d8-40b7-9227-6b1263aa006d None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:53.715745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc1602c-11d8-40b7-9227-6b1263aa006d None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:53.717488 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc1602c-11d8-40b7-9227-6b1263aa006d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:53.719495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 188/375] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:22:53 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:53.720110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc1602c-11d8-40b7-9227-6b1263aa006d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:53.738528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b14818-3d4d-4613-aa44-3dcb7fee89a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:53.739165 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b14818-3d4d-4613-aa44-3dcb7fee89a5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:53.739493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b14818-3d4d-4613-aa44-3dcb7fee89a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:53.739772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b14818-3d4d-4613-aa44-3dcb7fee89a5 None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:53.740972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b14818-3d4d-4613-aa44-3dcb7fee89a5 None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:53.742419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b14818-3d4d-4613-aa44-3dcb7fee89a5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:53.745261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 188/376] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:22:53 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:53.746446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb30637-f129-4bf4-838d-978f6cb703a5 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:53.746654 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb30637-f129-4bf4-838d-978f6cb703a5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:53.746932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb30637-f129-4bf4-838d-978f6cb703a5 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:53.748140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 189/377] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:53 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:53.748905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb30637-f129-4bf4-838d-978f6cb703a5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:53.771986 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:53.865554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc1602c-11d8-40b7-9227-6b1263aa006d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:53.889875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb30637-f129-4bf4-838d-978f6cb703a5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:53.908539 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189abb7-8b6b-4adb-80ed-33821d014242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:53.909412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189abb7-8b6b-4adb-80ed-33821d014242 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:53.909854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189abb7-8b6b-4adb-80ed-33821d014242 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:53.910708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189abb7-8b6b-4adb-80ed-33821d014242 None admin] PATH_INFO: `/v3/groups/18c3e22fdf9c477a9d0a6f0d44e4e828` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:53.916872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2189abb7-8b6b-4adb-80ed-33821d014242 None admin] RBAC: Authorizing `identity:get_group(group_id=18c3e22fdf9c477a9d0a6f0d44e4e828)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:53.918690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2189abb7-8b6b-4adb-80ed-33821d014242 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:53.921965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 189/378] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:22:53 2022] GET /identity/v3/groups/18c3e22fdf9c477a9d0a6f0d44e4e828 => generated 229 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:53.923117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189abb7-8b6b-4adb-80ed-33821d014242 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:53.933162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:53.934108 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ded04a1-adbf-4a9e-890c-15fb6c1cb8f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:53.934901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ded04a1-adbf-4a9e-890c-15fb6c1cb8f9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:53.935321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ded04a1-adbf-4a9e-890c-15fb6c1cb8f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:53.935744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ded04a1-adbf-4a9e-890c-15fb6c1cb8f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:53.937161 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ded04a1-adbf-4a9e-890c-15fb6c1cb8f9 None admin] RBAC: Adding query filter params (group.id=41b57b9e63f14cb3afba7976a393b32a, role.id=27476172dbb84c6e86b9070e02d06100, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:53.937473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ded04a1-adbf-4a9e-890c-15fb6c1cb8f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:53.939028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ded04a1-adbf-4a9e-890c-15fb6c1cb8f9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:53.947313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 190/379] 10.222.0.25 () {56 vars in 1376 bytes} [Mon Sep 26 01:22:53 2022] GET /identity/v3/role_assignments?group.id=41b57b9e63f14cb3afba7976a393b32a&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=27476172dbb84c6e86b9070e02d06100 => generated 260 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:53.949160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1475f44a-06ce-4704-bfa3-286197f388df None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:53.949448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1475f44a-06ce-4704-bfa3-286197f388df None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:53.950020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1475f44a-06ce-4704-bfa3-286197f388df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:53.976160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1475f44a-06ce-4704-bfa3-286197f388df None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:53.978339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1475f44a-06ce-4704-bfa3-286197f388df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:54.080652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2189abb7-8b6b-4adb-80ed-33821d014242 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:54.090072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 191/380] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:54.091619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1475f44a-06ce-4704-bfa3-286197f388df None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:54.111265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:54.154826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb99e1ab-9d70-408f-a422-91f02ab4766f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:54.155471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99e1ab-9d70-408f-a422-91f02ab4766f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:54.155756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99e1ab-9d70-408f-a422-91f02ab4766f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:54.156031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99e1ab-9d70-408f-a422-91f02ab4766f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:54.156815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb99e1ab-9d70-408f-a422-91f02ab4766f None admin] RBAC: Adding query filter params (group.id=41b57b9e63f14cb3afba7976a393b32a, role.id=48389faedba2470a885fae19084bb456, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:54.157123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb99e1ab-9d70-408f-a422-91f02ab4766f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:54.158675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb99e1ab-9d70-408f-a422-91f02ab4766f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:54.165979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 190/381] 10.222.0.25 () {56 vars in 1376 bytes} [Mon Sep 26 01:22:53 2022] GET /identity/v3/role_assignments?group.id=41b57b9e63f14cb3afba7976a393b32a&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=48389faedba2470a885fae19084bb456 => generated 260 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:54.253385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1475f44a-06ce-4704-bfa3-286197f388df None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:54.306241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c966f8-46cf-4de3-965c-13cf7a7a5e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:54.306880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c966f8-46cf-4de3-965c-13cf7a7a5e02 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:54.307162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c966f8-46cf-4de3-965c-13cf7a7a5e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:54.307438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c966f8-46cf-4de3-965c-13cf7a7a5e02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:54.308179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c966f8-46cf-4de3-965c-13cf7a7a5e02 None admin] RBAC: Adding query filter params (group.id=18c3e22fdf9c477a9d0a6f0d44e4e828, role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=7124d36dd410484faf5f0034e732bac5) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:54.308519 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c966f8-46cf-4de3-965c-13cf7a7a5e02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:54.310135 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c966f8-46cf-4de3-965c-13cf7a7a5e02 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:54.321913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 192/382] 10.222.0.25 () {56 vars in 1376 bytes} [Mon Sep 26 01:22:54 2022] GET /identity/v3/role_assignments?group.id=18c3e22fdf9c477a9d0a6f0d44e4e828&scope.project.id=7124d36dd410484faf5f0034e732bac5&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 260 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:54.386969 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caefb54c-33b2-4f3c-9220-76b211147169 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:54.387494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caefb54c-33b2-4f3c-9220-76b211147169 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:54.387494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caefb54c-33b2-4f3c-9220-76b211147169 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:54.388890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 191/383] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:54.394541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fbc27f-c46c-44d0-87a9-b85fc1b95227 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:54.394831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fbc27f-c46c-44d0-87a9-b85fc1b95227 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:54.395104 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fbc27f-c46c-44d0-87a9-b85fc1b95227 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:54.415065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-64fbc27f-c46c-44d0-87a9-b85fc1b95227 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:54.416444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-64fbc27f-c46c-44d0-87a9-b85fc1b95227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:54.466811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caefb54c-33b2-4f3c-9220-76b211147169 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:54.506211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:54.517502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 193/384] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:54.574765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:54.623002 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caefb54c-33b2-4f3c-9220-76b211147169 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:54.683121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03268568-ba37-4d81-af0c-ad6d160e20a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:54.683673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03268568-ba37-4d81-af0c-ad6d160e20a9 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:54.683950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03268568-ba37-4d81-af0c-ad6d160e20a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:54.684268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03268568-ba37-4d81-af0c-ad6d160e20a9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:54.708835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03268568-ba37-4d81-af0c-ad6d160e20a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-03268568-ba37-4d81-af0c-ad6d160e20a9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:54.714328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:54.715247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:54.716064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 192/385] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:54.723147 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fbc27f-c46c-44d0-87a9-b85fc1b95227 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:54.851867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fbc27f-c46c-44d0-87a9-b85fc1b95227 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:54.895577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276fc592-1238-4ceb-84cd-9f00e667cd60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:54.896190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276fc592-1238-4ceb-84cd-9f00e667cd60 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:54.896500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276fc592-1238-4ceb-84cd-9f00e667cd60 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:54.896773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276fc592-1238-4ceb-84cd-9f00e667cd60 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:54.897441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276fc592-1238-4ceb-84cd-9f00e667cd60 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:54.897746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276fc592-1238-4ceb-84cd-9f00e667cd60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:54.899114 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276fc592-1238-4ceb-84cd-9f00e667cd60 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:54.912254 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 194/386] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:54.918349 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03268568-ba37-4d81-af0c-ad6d160e20a9 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:54.937501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276fc592-1238-4ceb-84cd-9f00e667cd60 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:55.034996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:55.051703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03268568-ba37-4d81-af0c-ad6d160e20a9 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:55.082575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:55.094556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c358dc4a-0dd2-433f-bf64-537943ed61db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:55.095197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c358dc4a-0dd2-433f-bf64-537943ed61db None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:55.095475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c358dc4a-0dd2-433f-bf64-537943ed61db None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:55.095750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c358dc4a-0dd2-433f-bf64-537943ed61db None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:55.097634 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c358dc4a-0dd2-433f-bf64-537943ed61db None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:55.099067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c358dc4a-0dd2-433f-bf64-537943ed61db None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:55.100671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 193/387] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:22:54 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:55.109339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c358dc4a-0dd2-433f-bf64-537943ed61db None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:55.124656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276fc592-1238-4ceb-84cd-9f00e667cd60 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:55.194808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5ac44a-262b-4515-93d5-e780bdb4aa7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:55.195350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5ac44a-262b-4515-93d5-e780bdb4aa7e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:55.195629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5ac44a-262b-4515-93d5-e780bdb4aa7e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:55.195900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5ac44a-262b-4515-93d5-e780bdb4aa7e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:55.215589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5ac44a-262b-4515-93d5-e780bdb4aa7e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4c5ac44a-262b-4515-93d5-e780bdb4aa7e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:55.220945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:55.221842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:55.222626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 195/388] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:55.233336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c358dc4a-0dd2-433f-bf64-537943ed61db None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:55.245387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5ac44a-262b-4515-93d5-e780bdb4aa7e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:55.277174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dd04a3-c5b7-43b6-82ae-7e34bf3f1e3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:55.277819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dd04a3-c5b7-43b6-82ae-7e34bf3f1e3a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:55.278161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dd04a3-c5b7-43b6-82ae-7e34bf3f1e3a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:55.278357 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71dd04a3-c5b7-43b6-82ae-7e34bf3f1e3a None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:55.279624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dd04a3-c5b7-43b6-82ae-7e34bf3f1e3a None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:55.281110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71dd04a3-c5b7-43b6-82ae-7e34bf3f1e3a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:55.282732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 194/389] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:55 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:55.288535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dd04a3-c5b7-43b6-82ae-7e34bf3f1e3a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:55.369204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:55.408257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71dd04a3-c5b7-43b6-82ae-7e34bf3f1e3a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:55.465378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e686e4-e116-4562-8bb7-5ce7d5fd7164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:55.466044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e686e4-e116-4562-8bb7-5ce7d5fd7164 None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:55.466321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e686e4-e116-4562-8bb7-5ce7d5fd7164 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:55.466593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e686e4-e116-4562-8bb7-5ce7d5fd7164 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc/groups/41b57b9e63f14cb3afba7976a393b32a/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:55.469657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e686e4-e116-4562-8bb7-5ce7d5fd7164 None admin] RBAC: Authorizing `identity:create_grant(project_id=5f976380e1494db480dc80868d51a0bc, group_id=41b57b9e63f14cb3afba7976a393b32a, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:55.471091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e686e4-e116-4562-8bb7-5ce7d5fd7164 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:55.490224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 195/390] 10.222.0.25 () {58 vars in 1442 bytes} [Mon Sep 26 01:22:55 2022] PUT /identity/v3/projects/5f976380e1494db480dc80868d51a0bc/groups/41b57b9e63f14cb3afba7976a393b32a/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:22:55.490558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5ac44a-262b-4515-93d5-e780bdb4aa7e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:55.535532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:55.554876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9fe87d-b733-4cef-bbdd-7fc4cf073bff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:55.555505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9fe87d-b733-4cef-bbdd-7fc4cf073bff None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:55.556445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9fe87d-b733-4cef-bbdd-7fc4cf073bff None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:55.556727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9fe87d-b733-4cef-bbdd-7fc4cf073bff None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:55.557503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9fe87d-b733-4cef-bbdd-7fc4cf073bff None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:55.557810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9fe87d-b733-4cef-bbdd-7fc4cf073bff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:55.559232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9fe87d-b733-4cef-bbdd-7fc4cf073bff None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:55.573437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 196/391] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:55.579582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e686e4-e116-4562-8bb7-5ce7d5fd7164 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:55.747112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e686e4-e116-4562-8bb7-5ce7d5fd7164 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:55.779591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:55.791418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84068685-1a8a-4855-82b0-fc5bf314b4a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:55.792558 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84068685-1a8a-4855-82b0-fc5bf314b4a1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:55.793190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84068685-1a8a-4855-82b0-fc5bf314b4a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:55.793917 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84068685-1a8a-4855-82b0-fc5bf314b4a1 None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:55.795111 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84068685-1a8a-4855-82b0-fc5bf314b4a1 None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:55.796581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84068685-1a8a-4855-82b0-fc5bf314b4a1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:55.798456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 196/392] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:22:55 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:55.803778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9fe87d-b733-4cef-bbdd-7fc4cf073bff None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:55.949631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9fe87d-b733-4cef-bbdd-7fc4cf073bff None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:55.998073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aab81da-a8e5-43c9-9d6f-c06f8c413e89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:55.998632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aab81da-a8e5-43c9-9d6f-c06f8c413e89 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:55.998915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aab81da-a8e5-43c9-9d6f-c06f8c413e89 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:55.999190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aab81da-a8e5-43c9-9d6f-c06f8c413e89 None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:56.000361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aab81da-a8e5-43c9-9d6f-c06f8c413e89 None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:56.001807 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aab81da-a8e5-43c9-9d6f-c06f8c413e89 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:56.003433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 197/393] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:55 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:56.009895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84068685-1a8a-4855-82b0-fc5bf314b4a1 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:56.190298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84068685-1a8a-4855-82b0-fc5bf314b4a1 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:56.229961 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b16ed9-a69f-45c8-9b8f-a40801cdf52d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:56.230626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b16ed9-a69f-45c8-9b8f-a40801cdf52d None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:56.230905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b16ed9-a69f-45c8-9b8f-a40801cdf52d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:56.231235 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b16ed9-a69f-45c8-9b8f-a40801cdf52d None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc/groups/41b57b9e63f14cb3afba7976a393b32a/roles/48389faedba2470a885fae19084bb456` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:56.238114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b16ed9-a69f-45c8-9b8f-a40801cdf52d None admin] RBAC: Authorizing `identity:create_grant(project_id=5f976380e1494db480dc80868d51a0bc, group_id=41b57b9e63f14cb3afba7976a393b32a, role_id=48389faedba2470a885fae19084bb456)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:56.239560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b16ed9-a69f-45c8-9b8f-a40801cdf52d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:56.263512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 197/394] 10.222.0.25 () {58 vars in 1442 bytes} [Mon Sep 26 01:22:56 2022] PUT /identity/v3/projects/5f976380e1494db480dc80868d51a0bc/groups/41b57b9e63f14cb3afba7976a393b32a/roles/48389faedba2470a885fae19084bb456 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:56.933738 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5142adf-e810-4083-834d-a5bfd259d0aa None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:56.934253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5142adf-e810-4083-834d-a5bfd259d0aa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:56.934362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5142adf-e810-4083-834d-a5bfd259d0aa None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:56.935977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 198/395] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:56.941009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e43322-7563-46a6-86ef-c90190fa91bf None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:56.941438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e43322-7563-46a6-86ef-c90190fa91bf None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:56.941613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e43322-7563-46a6-86ef-c90190fa91bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:56.952521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54635b-5f6e-4b46-9873-cfaeac9f1943 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:56.952818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54635b-5f6e-4b46-9873-cfaeac9f1943 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:56.953073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54635b-5f6e-4b46-9873-cfaeac9f1943 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:56.954646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 199/396] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:56.959618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2fb782-072f-4e88-84a8-617ac8083cd9 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:56.959898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2fb782-072f-4e88-84a8-617ac8083cd9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:56.960170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2fb782-072f-4e88-84a8-617ac8083cd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:56.984159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-76e43322-7563-46a6-86ef-c90190fa91bf None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:56.985399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-76e43322-7563-46a6-86ef-c90190fa91bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:56.993059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fb2fb782-072f-4e88-84a8-617ac8083cd9 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:56.994416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fb2fb782-072f-4e88-84a8-617ac8083cd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:57.088184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:57.092731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:57.127161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 198/397] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:57.137728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 200/398] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:57.220952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1545dae5-25fc-4b19-9f78-522c9aab2538 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:57.221243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1545dae5-25fc-4b19-9f78-522c9aab2538 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:57.221500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1545dae5-25fc-4b19-9f78-522c9aab2538 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:57.223108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 199/399] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:57.229247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1a1789-dd6d-4c16-962f-d763ffcd8873 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:57.229533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1a1789-dd6d-4c16-962f-d763ffcd8873 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:57.229808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1a1789-dd6d-4c16-962f-d763ffcd8873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:57.256386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ef1a1789-dd6d-4c16-962f-d763ffcd8873 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:57.257577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ef1a1789-dd6d-4c16-962f-d763ffcd8873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:57.350988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 201/400] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:57.441256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1545dae5-25fc-4b19-9f78-522c9aab2538 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:57.481377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1a1789-dd6d-4c16-962f-d763ffcd8873 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:57.600652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1545dae5-25fc-4b19-9f78-522c9aab2538 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:57.639307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0755ba-8b24-47dd-be01-906861a9747a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:57.640391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0755ba-8b24-47dd-be01-906861a9747a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:57.640874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0755ba-8b24-47dd-be01-906861a9747a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:57.641365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0755ba-8b24-47dd-be01-906861a9747a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:57.661824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0755ba-8b24-47dd-be01-906861a9747a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ab0755ba-8b24-47dd-be01-906861a9747a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:57.673348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:57.674276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:57.675062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:57.676665 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1a1789-dd6d-4c16-962f-d763ffcd8873 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:57.677789 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 200/401] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:57.683123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0755ba-8b24-47dd-be01-906861a9747a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:57.708704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:57.737109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ffea43-35fc-40b7-bb04-6f2787c815a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:57.738117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ffea43-35fc-40b7-bb04-6f2787c815a1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:57.738497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ffea43-35fc-40b7-bb04-6f2787c815a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:57.738872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ffea43-35fc-40b7-bb04-6f2787c815a1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:57.750652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ffea43-35fc-40b7-bb04-6f2787c815a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:57.755219 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-54ffea43-35fc-40b7-bb04-6f2787c815a1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:57.757195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:57.758291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:57.759619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:57.765667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 202/402] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:57.765667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ffea43-35fc-40b7-bb04-6f2787c815a1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:57.826705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0755ba-8b24-47dd-be01-906861a9747a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:57.867655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ffea43-35fc-40b7-bb04-6f2787c815a1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:57.876460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ccade-2a35-4572-a88f-d5f4994c0226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:57.877296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ccade-2a35-4572-a88f-d5f4994c0226 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:57.877721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ccade-2a35-4572-a88f-d5f4994c0226 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:57.878560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ccade-2a35-4572-a88f-d5f4994c0226 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:57.879377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ccade-2a35-4572-a88f-d5f4994c0226 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:57.880220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ccade-2a35-4572-a88f-d5f4994c0226 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:57.882031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ccade-2a35-4572-a88f-d5f4994c0226 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:57.893526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 201/403] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:57.894707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ccade-2a35-4572-a88f-d5f4994c0226 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:57.909758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94e357c-eb95-4870-ab5d-7764ba62b313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:57.910540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94e357c-eb95-4870-ab5d-7764ba62b313 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:57.910888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94e357c-eb95-4870-ab5d-7764ba62b313 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:57.911236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94e357c-eb95-4870-ab5d-7764ba62b313 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:57.924907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94e357c-eb95-4870-ab5d-7764ba62b313 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d94e357c-eb95-4870-ab5d-7764ba62b313 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:57.931683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:57.932793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:22:57.933800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:22:57.934988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 203/404] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:22:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:57.940376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94e357c-eb95-4870-ab5d-7764ba62b313 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:57.981601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:58.026243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ccade-2a35-4572-a88f-d5f4994c0226 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:58.033521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:58.066154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94e357c-eb95-4870-ab5d-7764ba62b313 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:58.074221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac29bbf-6b79-49fc-9515-41c4af4e2d1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:58.074952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac29bbf-6b79-49fc-9515-41c4af4e2d1a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.075356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac29bbf-6b79-49fc-9515-41c4af4e2d1a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.075903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac29bbf-6b79-49fc-9515-41c4af4e2d1a None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.076554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac29bbf-6b79-49fc-9515-41c4af4e2d1a None admin] RBAC: Adding query filter params (name=member) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:58.076955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac29bbf-6b79-49fc-9515-41c4af4e2d1a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:58.078421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac29bbf-6b79-49fc-9515-41c4af4e2d1a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:58.092595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 202/405] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:58.092595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac29bbf-6b79-49fc-9515-41c4af4e2d1a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:58.112131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61211f07-87c1-46f3-9527-99652f67dc1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:58.113088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61211f07-87c1-46f3-9527-99652f67dc1e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.113545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61211f07-87c1-46f3-9527-99652f67dc1e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.113992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61211f07-87c1-46f3-9527-99652f67dc1e None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.115360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61211f07-87c1-46f3-9527-99652f67dc1e None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:58.117163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61211f07-87c1-46f3-9527-99652f67dc1e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:58.119726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 204/406] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:22:57 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:58.121056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61211f07-87c1-46f3-9527-99652f67dc1e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:58.235185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac29bbf-6b79-49fc-9515-41c4af4e2d1a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:58.246590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:58.252706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61211f07-87c1-46f3-9527-99652f67dc1e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:58.276966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5a97ab-a437-4609-a476-133fcf5450f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:58.278213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5a97ab-a437-4609-a476-133fcf5450f5 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.278944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5a97ab-a437-4609-a476-133fcf5450f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.279426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5a97ab-a437-4609-a476-133fcf5450f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.280931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5a97ab-a437-4609-a476-133fcf5450f5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:58.281418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5a97ab-a437-4609-a476-133fcf5450f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:58.283832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5a97ab-a437-4609-a476-133fcf5450f5 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:58.291743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fd6f7a-43c9-4f6e-b649-20f1bc38d841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:58.292073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fd6f7a-43c9-4f6e-b649-20f1bc38d841 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.292470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fd6f7a-43c9-4f6e-b649-20f1bc38d841 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.292813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fd6f7a-43c9-4f6e-b649-20f1bc38d841 None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.294006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fd6f7a-43c9-4f6e-b649-20f1bc38d841 None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:58.295504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fd6f7a-43c9-4f6e-b649-20f1bc38d841 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:58.296657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 203/407] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:58.298428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5a97ab-a437-4609-a476-133fcf5450f5 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:58.307224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 205/408] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:22:58 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:58.307224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fd6f7a-43c9-4f6e-b649-20f1bc38d841 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:58.339129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:58.411901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fd6f7a-43c9-4f6e-b649-20f1bc38d841 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:58.412807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5a97ab-a437-4609-a476-133fcf5450f5 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:58.447380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947ef12f-2d66-41bd-a4b5-905cca7da5c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:58.447598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47a532e-fdd2-41c6-9be5-afd42312386c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:58.448165 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47a532e-fdd2-41c6-9be5-afd42312386c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.448471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47a532e-fdd2-41c6-9be5-afd42312386c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.448745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47a532e-fdd2-41c6-9be5-afd42312386c None admin] PATH_INFO: `/v3/groups/18c3e22fdf9c477a9d0a6f0d44e4e828` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.448987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947ef12f-2d66-41bd-a4b5-905cca7da5c4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.449447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947ef12f-2d66-41bd-a4b5-905cca7da5c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.449867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47a532e-fdd2-41c6-9be5-afd42312386c None admin] RBAC: Authorizing `identity:get_group(group_id=18c3e22fdf9c477a9d0a6f0d44e4e828)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:58.450067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947ef12f-2d66-41bd-a4b5-905cca7da5c4 None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.451302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47a532e-fdd2-41c6-9be5-afd42312386c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:58.451518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947ef12f-2d66-41bd-a4b5-905cca7da5c4 None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:58.452881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 206/409] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:22:58 2022] GET /identity/v3/groups/18c3e22fdf9c477a9d0a6f0d44e4e828 => generated 229 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:58.453303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947ef12f-2d66-41bd-a4b5-905cca7da5c4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:58.453641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47a532e-fdd2-41c6-9be5-afd42312386c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:58.455249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 204/410] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:58 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:58.456847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8781f5ea-086f-4bf3-bfc6-c05b86b6f78f None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.457350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8781f5ea-086f-4bf3-bfc6-c05b86b6f78f None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.457786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8781f5ea-086f-4bf3-bfc6-c05b86b6f78f None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.463748 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 205/411] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:58 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:58.464792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8781f5ea-086f-4bf3-bfc6-c05b86b6f78f None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:58.496867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:58.566287 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47a532e-fdd2-41c6-9be5-afd42312386c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:58.570199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:58.575736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8781f5ea-086f-4bf3-bfc6-c05b86b6f78f None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:58.605441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b940acc-2e58-4c1e-8cc0-cd3b09ab47bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:58.606068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b940acc-2e58-4c1e-8cc0-cd3b09ab47bb None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.606411 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b940acc-2e58-4c1e-8cc0-cd3b09ab47bb None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.606682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b940acc-2e58-4c1e-8cc0-cd3b09ab47bb None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.607805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b940acc-2e58-4c1e-8cc0-cd3b09ab47bb None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:58.609294 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b940acc-2e58-4c1e-8cc0-cd3b09ab47bb None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:58.611043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 207/412] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:58 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:58.611848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b940acc-2e58-4c1e-8cc0-cd3b09ab47bb None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:58.618868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d17767-c2af-45e0-a50a-dc25903f13c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:58.619740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d17767-c2af-45e0-a50a-dc25903f13c8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.620021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d17767-c2af-45e0-a50a-dc25903f13c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.620297 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d17767-c2af-45e0-a50a-dc25903f13c8 None admin] PATH_INFO: `/v3/projects/7124d36dd410484faf5f0034e732bac5` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.621487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d17767-c2af-45e0-a50a-dc25903f13c8 None admin] RBAC: Authorizing `identity:get_project(project_id=7124d36dd410484faf5f0034e732bac5)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:58.622923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d17767-c2af-45e0-a50a-dc25903f13c8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:58.624920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 206/413] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:58 2022] GET /identity/v3/projects/7124d36dd410484faf5f0034e732bac5 => generated 344 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:22:58.626105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45054563-cc04-4a10-908c-f2c9e9e9ffd4 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.626392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45054563-cc04-4a10-908c-f2c9e9e9ffd4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.626666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45054563-cc04-4a10-908c-f2c9e9e9ffd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.645855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-45054563-cc04-4a10-908c-f2c9e9e9ffd4 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:58.647060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-45054563-cc04-4a10-908c-f2c9e9e9ffd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:58.720251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 207/414] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:22:58.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b940acc-2e58-4c1e-8cc0-cd3b09ab47bb None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:58.728757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45054563-cc04-4a10-908c-f2c9e9e9ffd4 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:58.752724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:58.760948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e84ca2-cb29-4d96-baaf-2cbb446de521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:58.761870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e84ca2-cb29-4d96-baaf-2cbb446de521 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.762221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e84ca2-cb29-4d96-baaf-2cbb446de521 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.762610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e84ca2-cb29-4d96-baaf-2cbb446de521 None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/groups/41b57b9e63f14cb3afba7976a393b32a/roles/48389faedba2470a885fae19084bb456` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.765735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e84ca2-cb29-4d96-baaf-2cbb446de521 None admin] RBAC: Authorizing `identity:create_grant(project_id=06f515d57f2143c8ad79c6d8440ceab3, group_id=41b57b9e63f14cb3afba7976a393b32a, role_id=48389faedba2470a885fae19084bb456)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:58.767830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e84ca2-cb29-4d96-baaf-2cbb446de521 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:58.793234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 208/415] 10.222.0.25 () {58 vars in 1442 bytes} [Mon Sep 26 01:22:58 2022] PUT /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/groups/41b57b9e63f14cb3afba7976a393b32a/roles/48389faedba2470a885fae19084bb456 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:58.794289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e84ca2-cb29-4d96-baaf-2cbb446de521 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:58.914556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:58.922203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45054563-cc04-4a10-908c-f2c9e9e9ffd4 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:58.965909 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0638c770-e7d8-41a2-93be-4c727b8f53fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:58.966594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0638c770-e7d8-41a2-93be-4c727b8f53fe None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:58.966877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0638c770-e7d8-41a2-93be-4c727b8f53fe None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:58.967155 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0638c770-e7d8-41a2-93be-4c727b8f53fe None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/groups/41b57b9e63f14cb3afba7976a393b32a/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:58.969669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0638c770-e7d8-41a2-93be-4c727b8f53fe None admin] RBAC: Authorizing `identity:create_grant(project_id=06f515d57f2143c8ad79c6d8440ceab3, group_id=41b57b9e63f14cb3afba7976a393b32a, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:58.971184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0638c770-e7d8-41a2-93be-4c727b8f53fe None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:58.991249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 208/416] 10.222.0.25 () {58 vars in 1442 bytes} [Mon Sep 26 01:22:58 2022] PUT /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3/groups/41b57b9e63f14cb3afba7976a393b32a/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:59.012216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0638c770-e7d8-41a2-93be-4c727b8f53fe None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:59.028375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e84ca2-cb29-4d96-baaf-2cbb446de521 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:59.075495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34e196d-8670-47ca-9f76-525855ea5384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:59.076630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e196d-8670-47ca-9f76-525855ea5384 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:59.076916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e196d-8670-47ca-9f76-525855ea5384 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:59.077188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34e196d-8670-47ca-9f76-525855ea5384 None admin] PATH_INFO: `/v3/projects/7124d36dd410484faf5f0034e732bac5/groups/18c3e22fdf9c477a9d0a6f0d44e4e828/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:59.079691 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34e196d-8670-47ca-9f76-525855ea5384 None admin] RBAC: Authorizing `identity:create_grant(project_id=7124d36dd410484faf5f0034e732bac5, group_id=18c3e22fdf9c477a9d0a6f0d44e4e828, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:59.081538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34e196d-8670-47ca-9f76-525855ea5384 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:59.098153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 209/417] 10.222.0.25 () {58 vars in 1442 bytes} [Mon Sep 26 01:22:58 2022] PUT /identity/v3/projects/7124d36dd410484faf5f0034e732bac5/groups/18c3e22fdf9c477a9d0a6f0d44e4e828/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:22:59.099434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b5dae-fa32-422b-81f6-d4b678207376 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:59.099719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b5dae-fa32-422b-81f6-d4b678207376 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:59.099984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b5dae-fa32-422b-81f6-d4b678207376 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:59.101656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 210/418] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:22:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:22:59.108282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0ff2a5-882d-4e6e-b610-64a8a999417d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:59.108609 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0ff2a5-882d-4e6e-b610-64a8a999417d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:59.108886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0ff2a5-882d-4e6e-b610-64a8a999417d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:59.129109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3a0ff2a5-882d-4e6e-b610-64a8a999417d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:22:59.130462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3a0ff2a5-882d-4e6e-b610-64a8a999417d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:22:59.153039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:59.155035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0638c770-e7d8-41a2-93be-4c727b8f53fe None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:59.198414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f885d7a5-c9ce-40ca-8e0e-7ca8fecd7075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:59.199516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f885d7a5-c9ce-40ca-8e0e-7ca8fecd7075 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:59.199991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f885d7a5-c9ce-40ca-8e0e-7ca8fecd7075 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:59.200527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f885d7a5-c9ce-40ca-8e0e-7ca8fecd7075 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:59.214389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f885d7a5-c9ce-40ca-8e0e-7ca8fecd7075 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-f885d7a5-c9ce-40ca-8e0e-7ca8fecd7075 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:59.220439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:59.221425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:22:59.222247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:59.225190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 209/419] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:22:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:22:59.232207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f885d7a5-c9ce-40ca-8e0e-7ca8fecd7075 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:59.248028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 211/420] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:22:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:22:59.253661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:59.350175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f885d7a5-c9ce-40ca-8e0e-7ca8fecd7075 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:59.388057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0695a73-884d-49a4-8afd-6b618383f920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:59.389082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0695a73-884d-49a4-8afd-6b618383f920 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:59.389555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0695a73-884d-49a4-8afd-6b618383f920 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:59.390020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0695a73-884d-49a4-8afd-6b618383f920 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:59.390905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0695a73-884d-49a4-8afd-6b618383f920 None admin] RBAC: Adding query filter params (name=member) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:22:59.391490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0695a73-884d-49a4-8afd-6b618383f920 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:59.393077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0695a73-884d-49a4-8afd-6b618383f920 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:59.402350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 210/421] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:22:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:22:59.408374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0ff2a5-882d-4e6e-b610-64a8a999417d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:59.529168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0ff2a5-882d-4e6e-b610-64a8a999417d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:59.566207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0695a73-884d-49a4-8afd-6b618383f920 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:59.580630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fc5f68-fabb-4998-afcb-1feb1cf5f873 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:59.580630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc5f68-fabb-4998-afcb-1feb1cf5f873 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:59.580630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc5f68-fabb-4998-afcb-1feb1cf5f873 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:59.580630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc5f68-fabb-4998-afcb-1feb1cf5f873 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:59.608654 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:59.658894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fc5f68-fabb-4998-afcb-1feb1cf5f873 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:22:59.660903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fc5f68-fabb-4998-afcb-1feb1cf5f873 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:22:59.741351 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:22:59.787853 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0695a73-884d-49a4-8afd-6b618383f920 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:22:59.833876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 212/422] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:22:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:22:59.840437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fc5f68-fabb-4998-afcb-1feb1cf5f873 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:59.865449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e93fcc-ec98-43e2-93fc-0b3993877578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:22:59.867304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e93fcc-ec98-43e2-93fc-0b3993877578 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:22:59.867826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e93fcc-ec98-43e2-93fc-0b3993877578 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:22:59.869100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e93fcc-ec98-43e2-93fc-0b3993877578 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:22:59.890477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e93fcc-ec98-43e2-93fc-0b3993877578 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-38e93fcc-ec98-43e2-93fc-0b3993877578 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:22:59.901607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:22:59.902541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:22:59.903325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:22:59.905989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 211/423] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:22:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 341 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:22:59.911780 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e93fcc-ec98-43e2-93fc-0b3993877578 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:22:59.953823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:00.042409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fc5f68-fabb-4998-afcb-1feb1cf5f873 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:00.115027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e93fcc-ec98-43e2-93fc-0b3993877578 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:00.116942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e665062-a026-4f4f-b609-873a60b3251a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:00.117571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e665062-a026-4f4f-b609-873a60b3251a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:00.117841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e665062-a026-4f4f-b609-873a60b3251a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:00.118103 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e665062-a026-4f4f-b609-873a60b3251a None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:00.119734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e665062-a026-4f4f-b609-873a60b3251a None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:00.122162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e665062-a026-4f4f-b609-873a60b3251a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:00.128178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 213/424] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:22:59 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:00.135989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e665062-a026-4f4f-b609-873a60b3251a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:00.184759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805e9fee-e022-4580-9c95-a1308396817c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:00.185433 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805e9fee-e022-4580-9c95-a1308396817c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:00.185652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805e9fee-e022-4580-9c95-a1308396817c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:00.185937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805e9fee-e022-4580-9c95-a1308396817c None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:00.186619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805e9fee-e022-4580-9c95-a1308396817c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:00.186921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805e9fee-e022-4580-9c95-a1308396817c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:00.189548 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805e9fee-e022-4580-9c95-a1308396817c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:00.203472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:00.211679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 212/425] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:22:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:00.220325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805e9fee-e022-4580-9c95-a1308396817c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:00.336989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e665062-a026-4f4f-b609-873a60b3251a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:00.363791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:00.377848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7034befd-7f6a-46b2-868e-1ad35555e28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:00.378903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7034befd-7f6a-46b2-868e-1ad35555e28e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:00.379291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7034befd-7f6a-46b2-868e-1ad35555e28e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:00.379694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7034befd-7f6a-46b2-868e-1ad35555e28e None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:00.380924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7034befd-7f6a-46b2-868e-1ad35555e28e None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:00.383228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7034befd-7f6a-46b2-868e-1ad35555e28e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:00.385815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 214/426] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:23:00 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:00.393398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7034befd-7f6a-46b2-868e-1ad35555e28e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:00.446458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805e9fee-e022-4580-9c95-a1308396817c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:00.507052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07bd19d-21ec-4a46-9c14-0063cfdad195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:00.507657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07bd19d-21ec-4a46-9c14-0063cfdad195 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:00.507934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07bd19d-21ec-4a46-9c14-0063cfdad195 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:00.508206 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07bd19d-21ec-4a46-9c14-0063cfdad195 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:00.585967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:00.589596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07bd19d-21ec-4a46-9c14-0063cfdad195 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:00.591889 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07bd19d-21ec-4a46-9c14-0063cfdad195 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:00.646346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7034befd-7f6a-46b2-868e-1ad35555e28e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:00.702918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 213/427] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 483 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:00.708980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07bd19d-21ec-4a46-9c14-0063cfdad195 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:00.741058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e0e23-db97-480b-bc97-94f903ba76c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:00.741678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e0e23-db97-480b-bc97-94f903ba76c6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:00.741962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e0e23-db97-480b-bc97-94f903ba76c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:00.742239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e0e23-db97-480b-bc97-94f903ba76c6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:00.742969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286e0e23-db97-480b-bc97-94f903ba76c6 None admin] RBAC: Adding query filter params (group.id=41b57b9e63f14cb3afba7976a393b32a, role.id=27476172dbb84c6e86b9070e02d06100, scope.project.id=5f976380e1494db480dc80868d51a0bc) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:00.743326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286e0e23-db97-480b-bc97-94f903ba76c6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:00.744881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286e0e23-db97-480b-bc97-94f903ba76c6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:00.756803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 215/428] 10.222.0.25 () {56 vars in 1376 bytes} [Mon Sep 26 01:23:00 2022] GET /identity/v3/role_assignments?group.id=41b57b9e63f14cb3afba7976a393b32a&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=27476172dbb84c6e86b9070e02d06100 => generated 611 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:00.816882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:00.820169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07bd19d-21ec-4a46-9c14-0063cfdad195 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:00.860404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e80eca3-9edd-4c7b-b785-557298d0753e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:00.861128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e80eca3-9edd-4c7b-b785-557298d0753e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:00.861499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e80eca3-9edd-4c7b-b785-557298d0753e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:00.861867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e80eca3-9edd-4c7b-b785-557298d0753e None admin] PATH_INFO: `/v3/projects/5f976380e1494db480dc80868d51a0bc` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:00.863100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e80eca3-9edd-4c7b-b785-557298d0753e None admin] RBAC: Authorizing `identity:get_project(project_id=5f976380e1494db480dc80868d51a0bc)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:00.864671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e80eca3-9edd-4c7b-b785-557298d0753e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:00.866484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 214/429] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:23:00 2022] GET /identity/v3/projects/5f976380e1494db480dc80868d51a0bc => generated 298 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:00.873520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e0e23-db97-480b-bc97-94f903ba76c6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:00.992122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:01.055561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e0e23-db97-480b-bc97-94f903ba76c6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:01.092103 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dce37b8-2079-4e02-8a3a-3bb8d2e60e77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:01.092764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dce37b8-2079-4e02-8a3a-3bb8d2e60e77 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:01.093050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dce37b8-2079-4e02-8a3a-3bb8d2e60e77 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:01.093335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dce37b8-2079-4e02-8a3a-3bb8d2e60e77 None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:01.094445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dce37b8-2079-4e02-8a3a-3bb8d2e60e77 None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:01.095885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dce37b8-2079-4e02-8a3a-3bb8d2e60e77 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:01.097469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 216/430] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:23:00 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:01.104210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e80eca3-9edd-4c7b-b785-557298d0753e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:01.213438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e80eca3-9edd-4c7b-b785-557298d0753e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:01.254506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416fe78c-edd7-4662-9ca7-5f945479290f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:01.255115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416fe78c-edd7-4662-9ca7-5f945479290f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:01.255399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416fe78c-edd7-4662-9ca7-5f945479290f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:01.255673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416fe78c-edd7-4662-9ca7-5f945479290f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:01.256503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416fe78c-edd7-4662-9ca7-5f945479290f None admin] RBAC: Adding query filter params (group.id=41b57b9e63f14cb3afba7976a393b32a, role.id=48389faedba2470a885fae19084bb456, scope.project.id=5f976380e1494db480dc80868d51a0bc) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:01.256816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416fe78c-edd7-4662-9ca7-5f945479290f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:01.258243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416fe78c-edd7-4662-9ca7-5f945479290f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:01.267352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 215/431] 10.222.0.25 () {56 vars in 1376 bytes} [Mon Sep 26 01:23:01 2022] GET /identity/v3/role_assignments?group.id=41b57b9e63f14cb3afba7976a393b32a&scope.project.id=5f976380e1494db480dc80868d51a0bc&role.id=48389faedba2470a885fae19084bb456 => generated 611 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:01.420590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ed350c-f8d2-4da5-8efd-3c1f74c51601 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:01.420933 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ed350c-f8d2-4da5-8efd-3c1f74c51601 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:01.421145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ed350c-f8d2-4da5-8efd-3c1f74c51601 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:01.422494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 217/432] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:01.428183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690b9b63-34b8-424c-a98c-d9aabed81f46 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:01.428532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690b9b63-34b8-424c-a98c-d9aabed81f46 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:01.428815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690b9b63-34b8-424c-a98c-d9aabed81f46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:01.453017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-690b9b63-34b8-424c-a98c-d9aabed81f46 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:01.454408 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-690b9b63-34b8-424c-a98c-d9aabed81f46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:01.492087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:01.539578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 216/433] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:01.593498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e900c3d5-72e7-4dd8-892f-5525acb58793 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:01.594143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e900c3d5-72e7-4dd8-892f-5525acb58793 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:01.594614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e900c3d5-72e7-4dd8-892f-5525acb58793 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:01.596411 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 218/434] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:01.601930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db673282-1189-4d28-90af-1f2eebbfc1e8 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:01.602450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db673282-1189-4d28-90af-1f2eebbfc1e8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:01.603536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db673282-1189-4d28-90af-1f2eebbfc1e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:01.626032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-db673282-1189-4d28-90af-1f2eebbfc1e8 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:01.627280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-db673282-1189-4d28-90af-1f2eebbfc1e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:01.708459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 217/435] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:01.749066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1811e2-b5e8-451f-a65a-615899e6dab1 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:01.749367 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1811e2-b5e8-451f-a65a-615899e6dab1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:01.749553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1811e2-b5e8-451f-a65a-615899e6dab1 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:01.751013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 219/436] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:01.756807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a242637-e37f-41a7-ab12-73b5abe2d832 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:01.756807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a242637-e37f-41a7-ab12-73b5abe2d832 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:01.757028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a242637-e37f-41a7-ab12-73b5abe2d832 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:01.776540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-6a242637-e37f-41a7-ab12-73b5abe2d832 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:01.778056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6a242637-e37f-41a7-ab12-73b5abe2d832 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:01.838127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1811e2-b5e8-451f-a65a-615899e6dab1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:01.874472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 218/437] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:01.955494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1811e2-b5e8-451f-a65a-615899e6dab1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:01.974806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a242637-e37f-41a7-ab12-73b5abe2d832 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:01.979870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:01.992283 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67bada8-8368-4703-9b7d-b7c1150ff843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:01.993001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67bada8-8368-4703-9b7d-b7c1150ff843 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:01.993369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67bada8-8368-4703-9b7d-b7c1150ff843 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:01.993599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67bada8-8368-4703-9b7d-b7c1150ff843 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:02.001600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:02.007513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67bada8-8368-4703-9b7d-b7c1150ff843 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b67bada8-8368-4703-9b7d-b7c1150ff843 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:02.014474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:02.015378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:02.016197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 220/438] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:23:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:02.021175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67bada8-8368-4703-9b7d-b7c1150ff843 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:02.090612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a242637-e37f-41a7-ab12-73b5abe2d832 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:02.127142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67bada8-8368-4703-9b7d-b7c1150ff843 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:02.129577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e8d6d1-bc9d-4c79-90cd-4a87d070ebd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:02.130201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e8d6d1-bc9d-4c79-90cd-4a87d070ebd2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:02.130480 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e8d6d1-bc9d-4c79-90cd-4a87d070ebd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:02.130746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e8d6d1-bc9d-4c79-90cd-4a87d070ebd2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:02.141450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e8d6d1-bc9d-4c79-90cd-4a87d070ebd2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-c8e8d6d1-bc9d-4c79-90cd-4a87d070ebd2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:02.146247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:02.147139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 26 01:23:02.147953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:02.152534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 219/439] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:02.153424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e8d6d1-bc9d-4c79-90cd-4a87d070ebd2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:02.164008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181ae26c-2630-4eb3-ab92-d81fffee88e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:02.164639 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ae26c-2630-4eb3-ab92-d81fffee88e9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:02.164917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ae26c-2630-4eb3-ab92-d81fffee88e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:02.165851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ae26c-2630-4eb3-ab92-d81fffee88e9 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:02.166519 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181ae26c-2630-4eb3-ab92-d81fffee88e9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:02.166819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181ae26c-2630-4eb3-ab92-d81fffee88e9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:02.168219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181ae26c-2630-4eb3-ab92-d81fffee88e9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:02.176098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 221/440] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:02.176920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181ae26c-2630-4eb3-ab92-d81fffee88e9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:02.224516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:02.236208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:02.254578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e8d6d1-bc9d-4c79-90cd-4a87d070ebd2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:02.279001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181ae26c-2630-4eb3-ab92-d81fffee88e9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:02.287896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecaf44f-ae2b-4e0c-91d6-7982abde5451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:02.288538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecaf44f-ae2b-4e0c-91d6-7982abde5451 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:02.288819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecaf44f-ae2b-4e0c-91d6-7982abde5451 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:02.289094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecaf44f-ae2b-4e0c-91d6-7982abde5451 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:02.299049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecaf44f-ae2b-4e0c-91d6-7982abde5451 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-5ecaf44f-ae2b-4e0c-91d6-7982abde5451 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:02.303859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:02.304957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 220/441] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:02.309405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecaf44f-ae2b-4e0c-91d6-7982abde5451 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:02.318726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc897210-055b-43b8-905e-d9657bbbd412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:02.319325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc897210-055b-43b8-905e-d9657bbbd412 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:02.319601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc897210-055b-43b8-905e-d9657bbbd412 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:02.319871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc897210-055b-43b8-905e-d9657bbbd412 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:02.320562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc897210-055b-43b8-905e-d9657bbbd412 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:02.320869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc897210-055b-43b8-905e-d9657bbbd412 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:02.322787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc897210-055b-43b8-905e-d9657bbbd412 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:02.330855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 222/442] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:23:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:02.331853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc897210-055b-43b8-905e-d9657bbbd412 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:02.415404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecaf44f-ae2b-4e0c-91d6-7982abde5451 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:02.429259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc897210-055b-43b8-905e-d9657bbbd412 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:02.436085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:02.449458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693bd7c3-0794-44f5-bf15-bcfbcb5379e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:02.450061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693bd7c3-0794-44f5-bf15-bcfbcb5379e8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:02.450339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693bd7c3-0794-44f5-bf15-bcfbcb5379e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:02.450612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693bd7c3-0794-44f5-bf15-bcfbcb5379e8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:02.463800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:02.464635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30c6c58-3408-4d14-a223-af314b0134f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:02.465234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c6c58-3408-4d14-a223-af314b0134f9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:02.465532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c6c58-3408-4d14-a223-af314b0134f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:02.465782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c6c58-3408-4d14-a223-af314b0134f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:02.466441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30c6c58-3408-4d14-a223-af314b0134f9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:02.466743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30c6c58-3408-4d14-a223-af314b0134f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:02.468137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30c6c58-3408-4d14-a223-af314b0134f9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:02.476046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 223/443] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:02.477016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30c6c58-3408-4d14-a223-af314b0134f9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:02.524653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693bd7c3-0794-44f5-bf15-bcfbcb5379e8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:02.526040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693bd7c3-0794-44f5-bf15-bcfbcb5379e8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:02.576434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30c6c58-3408-4d14-a223-af314b0134f9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:02.610703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef551a5-2523-4745-8d21-f91255e0a2a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:02.611323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef551a5-2523-4745-8d21-f91255e0a2a1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:02.611600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef551a5-2523-4745-8d21-f91255e0a2a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:02.611870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef551a5-2523-4745-8d21-f91255e0a2a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:02.625005 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 221/444] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:02.625845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693bd7c3-0794-44f5-bf15-bcfbcb5379e8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:02.650547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:02.668618 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:02.683908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef551a5-2523-4745-8d21-f91255e0a2a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:02.685388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef551a5-2523-4745-8d21-f91255e0a2a1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:02.726511 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693bd7c3-0794-44f5-bf15-bcfbcb5379e8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:02.759736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290496fd-464c-44db-8b90-6f50a2cbb0d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:02.760328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290496fd-464c-44db-8b90-6f50a2cbb0d1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:02.760632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290496fd-464c-44db-8b90-6f50a2cbb0d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:02.760904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290496fd-464c-44db-8b90-6f50a2cbb0d1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:02.782214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 224/445] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:02.782982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef551a5-2523-4745-8d21-f91255e0a2a1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:02.827078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290496fd-464c-44db-8b90-6f50a2cbb0d1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:02.828532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290496fd-464c-44db-8b90-6f50a2cbb0d1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:02.850599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:02.869906 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:02.884890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef551a5-2523-4745-8d21-f91255e0a2a1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:02.918403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f08fa-bcbd-441f-9375-8d5df7b06ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:02.919023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f08fa-bcbd-441f-9375-8d5df7b06ead None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:02.919304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f08fa-bcbd-441f-9375-8d5df7b06ead None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:02.919578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f08fa-bcbd-441f-9375-8d5df7b06ead None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:02.921666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79f08fa-bcbd-441f-9375-8d5df7b06ead None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:02.923122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79f08fa-bcbd-441f-9375-8d5df7b06ead None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:02.924760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 225/446] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:23:02 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:02.925517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f08fa-bcbd-441f-9375-8d5df7b06ead None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:02.927380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 222/447] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:02.932765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290496fd-464c-44db-8b90-6f50a2cbb0d1 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:03.025830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f08fa-bcbd-441f-9375-8d5df7b06ead None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:03.031305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290496fd-464c-44db-8b90-6f50a2cbb0d1 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:03.061394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72af425-3069-4f0c-b7f2-436291399b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:03.062018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72af425-3069-4f0c-b7f2-436291399b4e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:03.062299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72af425-3069-4f0c-b7f2-436291399b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:03.062574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72af425-3069-4f0c-b7f2-436291399b4e None admin] PATH_INFO: `/v3/projects/06f515d57f2143c8ad79c6d8440ceab3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:03.063684 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72af425-3069-4f0c-b7f2-436291399b4e None admin] RBAC: Authorizing `identity:get_project(project_id=06f515d57f2143c8ad79c6d8440ceab3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:03.065177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72af425-3069-4f0c-b7f2-436291399b4e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:03.066785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 226/448] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:23:02 2022] GET /identity/v3/projects/06f515d57f2143c8ad79c6d8440ceab3 => generated 302 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:03.067543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72af425-3069-4f0c-b7f2-436291399b4e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:03.068060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efe88f6-fcee-4555-bb9e-d2d3c05929cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:03.068707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efe88f6-fcee-4555-bb9e-d2d3c05929cc None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:03.068987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efe88f6-fcee-4555-bb9e-d2d3c05929cc None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:03.069258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efe88f6-fcee-4555-bb9e-d2d3c05929cc None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:03.070364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efe88f6-fcee-4555-bb9e-d2d3c05929cc None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:03.071879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:03.072740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efe88f6-fcee-4555-bb9e-d2d3c05929cc None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:03.076949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 223/449] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:23:02 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:03.076949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efe88f6-fcee-4555-bb9e-d2d3c05929cc None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:03.088904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:03.169995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72af425-3069-4f0c-b7f2-436291399b4e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:03.176140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efe88f6-fcee-4555-bb9e-d2d3c05929cc None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:03.203661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dbc68e-280c-46b1-b924-f84e56e550d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:03.204285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dbc68e-280c-46b1-b924-f84e56e550d6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:03.204594 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dbc68e-280c-46b1-b924-f84e56e550d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:03.204867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0dbc68e-280c-46b1-b924-f84e56e550d6 None admin] PATH_INFO: `/v3/projects/7124d36dd410484faf5f0034e732bac5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:03.206234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dbc68e-280c-46b1-b924-f84e56e550d6 None admin] RBAC: Authorizing `identity:get_project(project_id=7124d36dd410484faf5f0034e732bac5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:03.207667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0dbc68e-280c-46b1-b924-f84e56e550d6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:03.209296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 227/450] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:23:03 2022] GET /identity/v3/projects/7124d36dd410484faf5f0034e732bac5 => generated 344 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:03.210047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dbc68e-280c-46b1-b924-f84e56e550d6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:03.210149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e86cef-2d27-4f19-b6ee-3f37067ee6bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:03.210726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e86cef-2d27-4f19-b6ee-3f37067ee6bd None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:03.211009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e86cef-2d27-4f19-b6ee-3f37067ee6bd None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:03.211286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e86cef-2d27-4f19-b6ee-3f37067ee6bd None admin] PATH_INFO: `/v3/groups/41b57b9e63f14cb3afba7976a393b32a` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:03.212646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e86cef-2d27-4f19-b6ee-3f37067ee6bd None admin] RBAC: Authorizing `identity:get_group(group_id=41b57b9e63f14cb3afba7976a393b32a)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:03.214085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e86cef-2d27-4f19-b6ee-3f37067ee6bd None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:03.220561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 224/451] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:23:03 2022] GET /identity/v3/groups/41b57b9e63f14cb3afba7976a393b32a => generated 226 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:03.220561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e86cef-2d27-4f19-b6ee-3f37067ee6bd None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:03.279555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:03.296510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:03.309337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0dbc68e-280c-46b1-b924-f84e56e550d6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:03.318334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e86cef-2d27-4f19-b6ee-3f37067ee6bd None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:03.343749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8356b5fc-6446-4fc9-b548-8496a09e798e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:03.344336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8356b5fc-6446-4fc9-b548-8496a09e798e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:03.344636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8356b5fc-6446-4fc9-b548-8496a09e798e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:03.344908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8356b5fc-6446-4fc9-b548-8496a09e798e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:03.345620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8356b5fc-6446-4fc9-b548-8496a09e798e None admin] RBAC: Adding query filter params (group.id=41b57b9e63f14cb3afba7976a393b32a, role.id=48389faedba2470a885fae19084bb456, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:03.345924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8356b5fc-6446-4fc9-b548-8496a09e798e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:03.347341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8356b5fc-6446-4fc9-b548-8496a09e798e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:03.352082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2fe6a4-7f92-4400-bd25-94929e38402b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:03.353116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2fe6a4-7f92-4400-bd25-94929e38402b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:03.353258 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 228/452] 10.222.0.25 () {56 vars in 1376 bytes} [Mon Sep 26 01:23:03 2022] GET /identity/v3/role_assignments?group.id=41b57b9e63f14cb3afba7976a393b32a&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=48389faedba2470a885fae19084bb456 => generated 611 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:03.353356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2fe6a4-7f92-4400-bd25-94929e38402b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:03.353426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2fe6a4-7f92-4400-bd25-94929e38402b None admin] PATH_INFO: `/v3/groups/18c3e22fdf9c477a9d0a6f0d44e4e828` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:03.353662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8356b5fc-6446-4fc9-b548-8496a09e798e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:03.354614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2fe6a4-7f92-4400-bd25-94929e38402b None admin] RBAC: Authorizing `identity:get_group(group_id=18c3e22fdf9c477a9d0a6f0d44e4e828)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:03.356054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2fe6a4-7f92-4400-bd25-94929e38402b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:03.362487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 225/453] 10.222.0.25 () {56 vars in 1180 bytes} [Mon Sep 26 01:23:03 2022] GET /identity/v3/groups/18c3e22fdf9c477a9d0a6f0d44e4e828 => generated 229 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:03.363749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2fe6a4-7f92-4400-bd25-94929e38402b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:03.475267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8356b5fc-6446-4fc9-b548-8496a09e798e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:03.479597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2fe6a4-7f92-4400-bd25-94929e38402b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:03.513605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dfc384-5532-480e-9e02-1e3b5e8cbff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:03.514221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dfc384-5532-480e-9e02-1e3b5e8cbff5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:03.514501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dfc384-5532-480e-9e02-1e3b5e8cbff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:03.514774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dfc384-5532-480e-9e02-1e3b5e8cbff5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:03.515505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dfc384-5532-480e-9e02-1e3b5e8cbff5 None admin] RBAC: Adding query filter params (group.id=41b57b9e63f14cb3afba7976a393b32a, role.id=27476172dbb84c6e86b9070e02d06100, scope.project.id=06f515d57f2143c8ad79c6d8440ceab3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:03.515807 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dfc384-5532-480e-9e02-1e3b5e8cbff5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:03.516487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e85afb-8a47-41ab-b693-0b0e153606c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:03.517097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e85afb-8a47-41ab-b693-0b0e153606c7 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:03.517279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dfc384-5532-480e-9e02-1e3b5e8cbff5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:03.517666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e85afb-8a47-41ab-b693-0b0e153606c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:03.517945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e85afb-8a47-41ab-b693-0b0e153606c7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:03.518751 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e85afb-8a47-41ab-b693-0b0e153606c7 None admin] RBAC: Adding query filter params (group.id=18c3e22fdf9c477a9d0a6f0d44e4e828, role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=7124d36dd410484faf5f0034e732bac5) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:03.519279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e85afb-8a47-41ab-b693-0b0e153606c7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:03.520387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:03.521374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e85afb-8a47-41ab-b693-0b0e153606c7 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:03.525283 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 229/454] 10.222.0.25 () {56 vars in 1376 bytes} [Mon Sep 26 01:23:03 2022] GET /identity/v3/role_assignments?group.id=41b57b9e63f14cb3afba7976a393b32a&scope.project.id=06f515d57f2143c8ad79c6d8440ceab3&role.id=27476172dbb84c6e86b9070e02d06100 => generated 611 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:03.529165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 226/455] 10.222.0.25 () {56 vars in 1376 bytes} [Mon Sep 26 01:23:03 2022] GET /identity/v3/role_assignments?group.id=18c3e22fdf9c477a9d0a6f0d44e4e828&scope.project.id=7124d36dd410484faf5f0034e732bac5&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 611 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:06.474339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7779c023-eec6-4c36-b606-5841f99c3808 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:06.480727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7779c023-eec6-4c36-b606-5841f99c3808 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:06.481166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7779c023-eec6-4c36-b606-5841f99c3808 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:06.482691 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 230/456] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:06 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:06.488845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2231782-bc26-4492-aaf5-3cd863a67a34 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:06.489275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2231782-bc26-4492-aaf5-3cd863a67a34 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:06.489427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2231782-bc26-4492-aaf5-3cd863a67a34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:06.521355 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e2231782-bc26-4492-aaf5-3cd863a67a34 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:06.522577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e2231782-bc26-4492-aaf5-3cd863a67a34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:06.532706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:06.648578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 227/457] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:06.786155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5692dad-556b-460a-a888-a657b93332b4 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:06.786831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5692dad-556b-460a-a888-a657b93332b4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:06.787311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5692dad-556b-460a-a888-a657b93332b4 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:06.789123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 231/458] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:06.794953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965ff9e1-6963-404e-9edd-57d1557655e9 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:06.795506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965ff9e1-6963-404e-9edd-57d1557655e9 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:06.795999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965ff9e1-6963-404e-9edd-57d1557655e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:06.833339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-965ff9e1-6963-404e-9edd-57d1557655e9 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:06.834890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-965ff9e1-6963-404e-9edd-57d1557655e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:06.974004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 228/459] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:07.025071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5692dad-556b-460a-a888-a657b93332b4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:07.060222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33bfa1a-0eaa-49aa-96bc-c5a672161166 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:07.060542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33bfa1a-0eaa-49aa-96bc-c5a672161166 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:07.060825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33bfa1a-0eaa-49aa-96bc-c5a672161166 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:07.062758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 229/460] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:23:07.072380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82659cc-3275-43f2-bf1e-5269a2a0e874 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:07.072380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82659cc-3275-43f2-bf1e-5269a2a0e874 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:07.072380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82659cc-3275-43f2-bf1e-5269a2a0e874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:07.096581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e82659cc-3275-43f2-bf1e-5269a2a0e874 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:07.097795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e82659cc-3275-43f2-bf1e-5269a2a0e874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:07.194419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5692dad-556b-460a-a888-a657b93332b4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:07.230950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e6852c-6063-4d4e-a409-fa464aa7f272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:07.231578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e6852c-6063-4d4e-a409-fa464aa7f272 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:07.231856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e6852c-6063-4d4e-a409-fa464aa7f272 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:07.232129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e6852c-6063-4d4e-a409-fa464aa7f272 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:07.241695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:07.243970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e6852c-6063-4d4e-a409-fa464aa7f272 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:07.245558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e6852c-6063-4d4e-a409-fa464aa7f272 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:07.249618 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 230/461] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-34e6852c-6063-4d4e-a409-fa464aa7f272 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:07.270154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:07.271119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:07.272648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:07.272648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:07.272648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:07.272648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 232/462] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:07.278169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82659cc-3275-43f2-bf1e-5269a2a0e874 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:07.342292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e6852c-6063-4d4e-a409-fa464aa7f272 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:07.438722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:07.454923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82659cc-3275-43f2-bf1e-5269a2a0e874 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:07.498569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f731a0-9ac5-4a4e-8d44-f5b8e9549036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:07.499254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f731a0-9ac5-4a4e-8d44-f5b8e9549036 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:07.499466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e6852c-6063-4d4e-a409-fa464aa7f272 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:07.499872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f731a0-9ac5-4a4e-8d44-f5b8e9549036 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:07.500151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f731a0-9ac5-4a4e-8d44-f5b8e9549036 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:07.501028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f731a0-9ac5-4a4e-8d44-f5b8e9549036 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:07.501628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f731a0-9ac5-4a4e-8d44-f5b8e9549036 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:07.503304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f731a0-9ac5-4a4e-8d44-f5b8e9549036 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:07.517012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 231/463] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:07.518593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c472a57f-62cf-441c-84e0-82bbc3c3d422 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:07.519065 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c472a57f-62cf-441c-84e0-82bbc3c3d422 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:07.519521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c472a57f-62cf-441c-84e0-82bbc3c3d422 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:07.525212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 232/464] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:07.526553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c472a57f-62cf-441c-84e0-82bbc3c3d422 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:07.541669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e763d259-6ba3-4b37-8fd1-71a301e2ff4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:07.542448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e763d259-6ba3-4b37-8fd1-71a301e2ff4e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:07.542924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e763d259-6ba3-4b37-8fd1-71a301e2ff4e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:07.543395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e763d259-6ba3-4b37-8fd1-71a301e2ff4e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:07.555744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e763d259-6ba3-4b37-8fd1-71a301e2ff4e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:07.557823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e763d259-6ba3-4b37-8fd1-71a301e2ff4e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e763d259-6ba3-4b37-8fd1-71a301e2ff4e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:07.574196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:07.575160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:07.576031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:07.576031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:07.576031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:07.576031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 233/465] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:07.576998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaf8a15-2d83-4467-9493-dbfcd6a23cbb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:07.576998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaf8a15-2d83-4467-9493-dbfcd6a23cbb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:07.576998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaf8a15-2d83-4467-9493-dbfcd6a23cbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:07.595018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:07.595454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1eaf8a15-2d83-4467-9493-dbfcd6a23cbb None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:07.596676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1eaf8a15-2d83-4467-9493-dbfcd6a23cbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:07.642783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c472a57f-62cf-441c-84e0-82bbc3c3d422 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:07.672395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 234/466] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:07.673746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eaf8a15-2d83-4467-9493-dbfcd6a23cbb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:07.692835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f387c4a-edb1-47a7-a7c3-d824ec4e41d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:07.694547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f387c4a-edb1-47a7-a7c3-d824ec4e41d0 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:07.695785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f387c4a-edb1-47a7-a7c3-d824ec4e41d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:07.696519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f387c4a-edb1-47a7-a7c3-d824ec4e41d0 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:07.699113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f387c4a-edb1-47a7-a7c3-d824ec4e41d0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:07.700643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f387c4a-edb1-47a7-a7c3-d824ec4e41d0 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:07.716067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:07.735820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 233/467] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:23:07 2022] POST /identity/v3/users => generated 246 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:07.741593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f387c4a-edb1-47a7-a7c3-d824ec4e41d0 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:07.851960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eaf8a15-2d83-4467-9493-dbfcd6a23cbb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:07.923982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb97487-e6ac-4c52-86c5-7b8598e91eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:07.925034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb97487-e6ac-4c52-86c5-7b8598e91eb4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:07.925506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb97487-e6ac-4c52-86c5-7b8598e91eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:07.925953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb97487-e6ac-4c52-86c5-7b8598e91eb4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:07.943545 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f387c4a-edb1-47a7-a7c3-d824ec4e41d0 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:07.949204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb97487-e6ac-4c52-86c5-7b8598e91eb4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:07.955458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb97487-e6ac-4c52-86c5-7b8598e91eb4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:07.967302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-afb97487-e6ac-4c52-86c5-7b8598e91eb4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:07.976383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:07.977318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:07.978621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:07.978621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:07.978621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:07.978621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 235/468] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:07.981681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb97487-e6ac-4c52-86c5-7b8598e91eb4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:08.000245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b60f0-3581-40d4-8970-f470fe40dade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:08.001072 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b60f0-3581-40d4-8970-f470fe40dade None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:08.001359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b60f0-3581-40d4-8970-f470fe40dade None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:08.001636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b60f0-3581-40d4-8970-f470fe40dade None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:08.002324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110b60f0-3581-40d4-8970-f470fe40dade None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:08.002630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110b60f0-3581-40d4-8970-f470fe40dade None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:08.004056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110b60f0-3581-40d4-8970-f470fe40dade None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:08.019706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 234/469] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:08.020651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b60f0-3581-40d4-8970-f470fe40dade None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:08.094751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:08.120195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb97487-e6ac-4c52-86c5-7b8598e91eb4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:08.154621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b60f0-3581-40d4-8970-f470fe40dade None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:08.159318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3eec35d-c5f6-49e2-be4a-17e3c0888fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:08.159950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3eec35d-c5f6-49e2-be4a-17e3c0888fff None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:08.160231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3eec35d-c5f6-49e2-be4a-17e3c0888fff None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:08.160536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3eec35d-c5f6-49e2-be4a-17e3c0888fff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:08.173292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3eec35d-c5f6-49e2-be4a-17e3c0888fff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:08.174752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3eec35d-c5f6-49e2-be4a-17e3c0888fff None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e3eec35d-c5f6-49e2-be4a-17e3c0888fff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:08.190073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:08.191006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:08.191858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:08.191858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:08.191858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:08.191858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 236/470] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:08.192083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3eec35d-c5f6-49e2-be4a-17e3c0888fff None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:08.199260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f17a35-10c2-4d60-b8e1-ba9580c7459c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:08.199878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f17a35-10c2-4d60-b8e1-ba9580c7459c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:08.200161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f17a35-10c2-4d60-b8e1-ba9580c7459c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:08.200468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f17a35-10c2-4d60-b8e1-ba9580c7459c None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:08.201125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f17a35-10c2-4d60-b8e1-ba9580c7459c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:08.201430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f17a35-10c2-4d60-b8e1-ba9580c7459c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:08.202806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f17a35-10c2-4d60-b8e1-ba9580c7459c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:08.215032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 235/471] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:08.216601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba1106e-f559-4ffd-8b3a-c017b5038ada None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:08.217110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba1106e-f559-4ffd-8b3a-c017b5038ada None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:08.217597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba1106e-f559-4ffd-8b3a-c017b5038ada None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:08.223114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 236/472] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:08.223114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba1106e-f559-4ffd-8b3a-c017b5038ada None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:08.262068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:08.307075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3eec35d-c5f6-49e2-be4a-17e3c0888fff None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:08.332974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba1106e-f559-4ffd-8b3a-c017b5038ada None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:08.344614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4c513a-807c-48c0-bdcd-acee5e929f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:08.345067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4c513a-807c-48c0-bdcd-acee5e929f80 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:08.345344 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4c513a-807c-48c0-bdcd-acee5e929f80 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:08.345607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4c513a-807c-48c0-bdcd-acee5e929f80 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:08.346310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4c513a-807c-48c0-bdcd-acee5e929f80 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:08.348061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4c513a-807c-48c0-bdcd-acee5e929f80 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:08.358987 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:08.369745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038e0fd0-4841-4f77-8edb-6b82afa13ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:08.370468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e0fd0-4841-4f77-8edb-6b82afa13ba8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:08.370687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e0fd0-4841-4f77-8edb-6b82afa13ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:08.370953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e0fd0-4841-4f77-8edb-6b82afa13ba8 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:08.371589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038e0fd0-4841-4f77-8edb-6b82afa13ba8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:08.371887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038e0fd0-4841-4f77-8edb-6b82afa13ba8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:08.380502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 237/473] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:23:08 2022] POST /identity/v3/users => generated 248 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:08.380698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038e0fd0-4841-4f77-8edb-6b82afa13ba8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:08.380802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4c513a-807c-48c0-bdcd-acee5e929f80 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:08.396426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 237/474] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:08.398557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6714c294-7919-424b-9cec-f914bbd6bd96 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:08.398848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6714c294-7919-424b-9cec-f914bbd6bd96 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:08.399126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6714c294-7919-424b-9cec-f914bbd6bd96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:08.420664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-6714c294-7919-424b-9cec-f914bbd6bd96 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:08.422226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6714c294-7919-424b-9cec-f914bbd6bd96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:08.510368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 238/475] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:08.511367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6714c294-7919-424b-9cec-f914bbd6bd96 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:08.513562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4c513a-807c-48c0-bdcd-acee5e929f80 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:08.552070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a073e1-93bd-4185-a53d-51fde1d2adc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:08.553137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a073e1-93bd-4185-a53d-51fde1d2adc7 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:08.553629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a073e1-93bd-4185-a53d-51fde1d2adc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:08.554111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a073e1-93bd-4185-a53d-51fde1d2adc7 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:08.555014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a073e1-93bd-4185-a53d-51fde1d2adc7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:08.556765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a073e1-93bd-4185-a53d-51fde1d2adc7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:08.569426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:08.595009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 238/476] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:23:08 2022] POST /identity/v3/users => generated 248 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:08.638661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6714c294-7919-424b-9cec-f914bbd6bd96 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:08.679548 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99a7843-1ffa-46f9-88ae-efc7c9436b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:08.680172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99a7843-1ffa-46f9-88ae-efc7c9436b53 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:08.680482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99a7843-1ffa-46f9-88ae-efc7c9436b53 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:08.681291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99a7843-1ffa-46f9-88ae-efc7c9436b53 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:08.681993 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99a7843-1ffa-46f9-88ae-efc7c9436b53 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:08.683454 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99a7843-1ffa-46f9-88ae-efc7c9436b53 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:08.709518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 239/477] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:23:08 2022] POST /identity/v3/users => generated 249 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:08.872269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a073e1-93bd-4185-a53d-51fde1d2adc7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:08.927067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:08.982401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a073e1-93bd-4185-a53d-51fde1d2adc7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:09.020037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ad51bc-0e7a-4cd0-a947-699fab8b3e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:09.020818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ad51bc-0e7a-4cd0-a947-699fab8b3e2e None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:09.021156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ad51bc-0e7a-4cd0-a947-699fab8b3e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:09.021418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ad51bc-0e7a-4cd0-a947-699fab8b3e2e None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:09.022086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ad51bc-0e7a-4cd0-a947-699fab8b3e2e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:09.023456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ad51bc-0e7a-4cd0-a947-699fab8b3e2e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:09.032243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.sql.core [None req-e3ad51bc-0e7a-4cd0-a947-699fab8b3e2e None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 26 01:23:09.032243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 26 01:23:09.032243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [parameters: {'id': '9c70f7f1fd5b441cbdb99a87820b4889', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 26 01:23:09.032243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81423) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e3ad51bc-0e7a-4cd0-a947-699fab8b3e2e None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result = self._query(query) Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application conn.query(q) Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result.read() Sep 26 01:23:09.049567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return ref.to_dict() Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 26 01:23:09.050495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application next(self.gen) Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application yield resource Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application next(self.gen) Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.session.rollback() Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.force_reraise() Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise self.value Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application session.commit() Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 26 01:23:09.051524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._prepare_impl() Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.session.flush() Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._flush(objects) Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application compat.raise_( Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application flush_context.execute() Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rec.execute(self) Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 26 01:23:09.052401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 26 01:23:09.053277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result = self._query(query) Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application conn.query(q) Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result.read() Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 26 01:23:09.054572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [parameters: {'id': '9c70f7f1fd5b441cbdb99a87820b4889', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:09.055368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 26 01:23:09.058482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:09.060565 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 239/478] 10.222.0.25 () {60 vars in 1131 bytes} [Mon Sep 26 01:23:08 2022] POST /identity/v3/roles => generated 144 bytes in 179 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:09.060706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99a7843-1ffa-46f9-88ae-efc7c9436b53 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:09.150740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:09.175179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99a7843-1ffa-46f9-88ae-efc7c9436b53 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:09.218538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ac2548-b84c-41ba-867d-38370a83f75d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:09.219179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ac2548-b84c-41ba-867d-38370a83f75d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:09.219455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ac2548-b84c-41ba-867d-38370a83f75d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:09.219726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ac2548-b84c-41ba-867d-38370a83f75d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:09.230606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ac2548-b84c-41ba-867d-38370a83f75d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-a4ac2548-b84c-41ba-867d-38370a83f75d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:09.235967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:09.237862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:09.238980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 240/479] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:23:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:09.242849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ad51bc-0e7a-4cd0-a947-699fab8b3e2e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:09.401515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ad51bc-0e7a-4cd0-a947-699fab8b3e2e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:09.425471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:09.452053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55769de-8a9f-4fba-806e-6ece2d4a53b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:09.452681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55769de-8a9f-4fba-806e-6ece2d4a53b0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:09.452958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55769de-8a9f-4fba-806e-6ece2d4a53b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:09.453228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55769de-8a9f-4fba-806e-6ece2d4a53b0 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:09.453896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55769de-8a9f-4fba-806e-6ece2d4a53b0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:09.454198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55769de-8a9f-4fba-806e-6ece2d4a53b0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:09.455568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55769de-8a9f-4fba-806e-6ece2d4a53b0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:09.471867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 240/480] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:09 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:10.624717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d694f59-80ff-4ab7-83e5-5c62f2832214 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:10.625253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d694f59-80ff-4ab7-83e5-5c62f2832214 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:10.625253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d694f59-80ff-4ab7-83e5-5c62f2832214 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:10.626793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 241/481] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:10.632161 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda3e9a1-cca4-4072-b8a4-ffb60090fee1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:10.632589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda3e9a1-cca4-4072-b8a4-ffb60090fee1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:10.632776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda3e9a1-cca4-4072-b8a4-ffb60090fee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:10.660551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bda3e9a1-cca4-4072-b8a4-ffb60090fee1 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:10.661741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bda3e9a1-cca4-4072-b8a4-ffb60090fee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:10.752818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 241/482] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:23:11.136447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e3b32-8d38-4ab2-ace2-b3cfb3121644 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:11.136769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e3b32-8d38-4ab2-ace2-b3cfb3121644 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:11.136995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e3b32-8d38-4ab2-ace2-b3cfb3121644 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:11.138569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 242/483] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:11.143491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17115984-a6e1-4d77-b05e-eafe7e410775 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:11.143777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17115984-a6e1-4d77-b05e-eafe7e410775 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:11.144045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17115984-a6e1-4d77-b05e-eafe7e410775 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:11.146377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e3b32-8d38-4ab2-ace2-b3cfb3121644 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:11.185270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-17115984-a6e1-4d77-b05e-eafe7e410775 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:11.186486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-17115984-a6e1-4d77-b05e-eafe7e410775 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:11.325241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 242/484] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:11.330972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e3b32-8d38-4ab2-ace2-b3cfb3121644 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:11.338804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ebed9d-6067-4429-9ade-4977f9f24dc9 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:11.339381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ebed9d-6067-4429-9ade-4977f9f24dc9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:11.340297 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ebed9d-6067-4429-9ade-4977f9f24dc9 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:11.342495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 243/485] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:23:11.347920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b85173-b733-43f4-bd5a-522bab9032b8 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:11.348481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b85173-b733-43f4-bd5a-522bab9032b8 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:11.348962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b85173-b733-43f4-bd5a-522bab9032b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:11.371073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e8b85173-b733-43f4-bd5a-522bab9032b8 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:11.372045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90592810-947b-42cb-b1c5-86763b8074ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:11.372721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90592810-947b-42cb-b1c5-86763b8074ea None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:11.372996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90592810-947b-42cb-b1c5-86763b8074ea None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:11.373264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90592810-947b-42cb-b1c5-86763b8074ea None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:11.373638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e8b85173-b733-43f4-bd5a-522bab9032b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:11.389338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90592810-947b-42cb-b1c5-86763b8074ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-90592810-947b-42cb-b1c5-86763b8074ea None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:11.395209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:11.396238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:11.397161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:11.397773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 243/486] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:11.405058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90592810-947b-42cb-b1c5-86763b8074ea None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:11.408021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:11.416864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:11.470586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 244/487] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:23:11.489133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e650c-6d1b-4d36-83b7-d7248a9203d9 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:11.489670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e650c-6d1b-4d36-83b7-d7248a9203d9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:11.490099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e650c-6d1b-4d36-83b7-d7248a9203d9 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:11.491831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 245/488] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:23:11.500328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10041673-0310-412a-abfd-efd97be254c2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:11.500861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10041673-0310-412a-abfd-efd97be254c2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:11.501303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10041673-0310-412a-abfd-efd97be254c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:11.541642 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-10041673-0310-412a-abfd-efd97be254c2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:11.545136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-10041673-0310-412a-abfd-efd97be254c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:11.550853 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90592810-947b-42cb-b1c5-86763b8074ea None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:11.622356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a108d9-a9e0-4814-bf85-cff3270235bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:11.623232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a108d9-a9e0-4814-bf85-cff3270235bb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:11.623681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a108d9-a9e0-4814-bf85-cff3270235bb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:11.624143 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a108d9-a9e0-4814-bf85-cff3270235bb None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:11.625004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a108d9-a9e0-4814-bf85-cff3270235bb None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:11.625902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a108d9-a9e0-4814-bf85-cff3270235bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:11.627455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a108d9-a9e0-4814-bf85-cff3270235bb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:11.646846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 244/489] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:11.653276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a108d9-a9e0-4814-bf85-cff3270235bb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:11.678951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 246/490] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:11.723131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:11.725753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10041673-0310-412a-abfd-efd97be254c2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:11.779393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a108d9-a9e0-4814-bf85-cff3270235bb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:11.817300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad334b6-e2de-40d2-a5bb-02439b46dd86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:11.817971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad334b6-e2de-40d2-a5bb-02439b46dd86 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:11.818297 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad334b6-e2de-40d2-a5bb-02439b46dd86 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:11.818567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad334b6-e2de-40d2-a5bb-02439b46dd86 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:11.830762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad334b6-e2de-40d2-a5bb-02439b46dd86 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:11.832760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad334b6-e2de-40d2-a5bb-02439b46dd86 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-fad334b6-e2de-40d2-a5bb-02439b46dd86 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:11.849472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:11.850836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:11.851907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:11.851907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:11.851907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:11.851907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 245/491] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:11.851907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad334b6-e2de-40d2-a5bb-02439b46dd86 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:11.880911 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:11.891510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10041673-0310-412a-abfd-efd97be254c2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:11.929728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d80a26c-58b6-4dfa-a458-9644793973bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:11.930414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d80a26c-58b6-4dfa-a458-9644793973bd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:11.930775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d80a26c-58b6-4dfa-a458-9644793973bd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:11.931196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d80a26c-58b6-4dfa-a458-9644793973bd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:11.941940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d80a26c-58b6-4dfa-a458-9644793973bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-1d80a26c-58b6-4dfa-a458-9644793973bd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:11.947316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:11.948233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:11.949106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 247/492] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:11.949772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d80a26c-58b6-4dfa-a458-9644793973bd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:11.984415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad334b6-e2de-40d2-a5bb-02439b46dd86 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.003130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:12.025695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49538272-448a-4dac-bc9e-b40b062f8619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.026328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49538272-448a-4dac-bc9e-b40b062f8619 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.026611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49538272-448a-4dac-bc9e-b40b062f8619 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.026886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49538272-448a-4dac-bc9e-b40b062f8619 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.039290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49538272-448a-4dac-bc9e-b40b062f8619 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-49538272-448a-4dac-bc9e-b40b062f8619 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:12.045007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:12.045903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 246/493] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:12.049312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49538272-448a-4dac-bc9e-b40b062f8619 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.069996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d80a26c-58b6-4dfa-a458-9644793973bd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.106875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c7d8ae-83bb-4a16-ad00-ce80c960487f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.107470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c7d8ae-83bb-4a16-ad00-ce80c960487f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.107744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c7d8ae-83bb-4a16-ad00-ce80c960487f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.108010 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c7d8ae-83bb-4a16-ad00-ce80c960487f None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.108679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c7d8ae-83bb-4a16-ad00-ce80c960487f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:12.108975 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c7d8ae-83bb-4a16-ad00-ce80c960487f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:12.110349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c7d8ae-83bb-4a16-ad00-ce80c960487f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:12.128277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 248/494] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:12.129143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c7d8ae-83bb-4a16-ad00-ce80c960487f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.150996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:12.171067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49538272-448a-4dac-bc9e-b40b062f8619 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.207500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcad9f76-4f44-4b6d-b2e0-edf6953c9aed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.208103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcad9f76-4f44-4b6d-b2e0-edf6953c9aed None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.208414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcad9f76-4f44-4b6d-b2e0-edf6953c9aed None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.208691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcad9f76-4f44-4b6d-b2e0-edf6953c9aed None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.209353 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcad9f76-4f44-4b6d-b2e0-edf6953c9aed None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:12.209655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcad9f76-4f44-4b6d-b2e0-edf6953c9aed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:12.211026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcad9f76-4f44-4b6d-b2e0-edf6953c9aed None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:12.221179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 247/495] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:12.222465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcad9f76-4f44-4b6d-b2e0-edf6953c9aed None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.236463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c7d8ae-83bb-4a16-ad00-ce80c960487f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.269534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:12.274881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6821f0-6343-4c62-84fa-22b26d075730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.275758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6821f0-6343-4c62-84fa-22b26d075730 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.276035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6821f0-6343-4c62-84fa-22b26d075730 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.276314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6821f0-6343-4c62-84fa-22b26d075730 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.286869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6821f0-6343-4c62-84fa-22b26d075730 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-dd6821f0-6343-4c62-84fa-22b26d075730 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:12.292158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:12.293176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:12.293985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 249/496] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:12.294635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6821f0-6343-4c62-84fa-22b26d075730 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.340645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcad9f76-4f44-4b6d-b2e0-edf6953c9aed None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.383368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:12.384547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221d3b52-d55e-4556-851a-7363e633f263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.385971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221d3b52-d55e-4556-851a-7363e633f263 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.386365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221d3b52-d55e-4556-851a-7363e633f263 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.386827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221d3b52-d55e-4556-851a-7363e633f263 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.387875 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221d3b52-d55e-4556-851a-7363e633f263 None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:12.388195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221d3b52-d55e-4556-851a-7363e633f263 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:12.389623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221d3b52-d55e-4556-851a-7363e633f263 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:12.398292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 248/497] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:12.400102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221d3b52-d55e-4556-851a-7363e633f263 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.410685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6821f0-6343-4c62-84fa-22b26d075730 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.446854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fc29ab-81bf-48b6-9246-fe59915d64f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.447493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fc29ab-81bf-48b6-9246-fe59915d64f1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.447769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fc29ab-81bf-48b6-9246-fe59915d64f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.448037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fc29ab-81bf-48b6-9246-fe59915d64f1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.455583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fc29ab-81bf-48b6-9246-fe59915d64f1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:12.457408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fc29ab-81bf-48b6-9246-fe59915d64f1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-24fc29ab-81bf-48b6-9246-fe59915d64f1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:12.472007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:12.472990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 250/498] 10.222.0.25 () {56 vars in 1093 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/users/nova => generated 82 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:12.473880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fc29ab-81bf-48b6-9246-fe59915d64f1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.515034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:12.518050 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221d3b52-d55e-4556-851a-7363e633f263 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.552095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5277c9ad-bbe4-46d8-9429-c9c998e8b83a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.552746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5277c9ad-bbe4-46d8-9429-c9c998e8b83a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.553028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5277c9ad-bbe4-46d8-9429-c9c998e8b83a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.553300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5277c9ad-bbe4-46d8-9429-c9c998e8b83a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.563623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5277c9ad-bbe4-46d8-9429-c9c998e8b83a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:12.565100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5277c9ad-bbe4-46d8-9429-c9c998e8b83a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:12.575002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fc29ab-81bf-48b6-9246-fe59915d64f1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-5277c9ad-bbe4-46d8-9429-c9c998e8b83a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:12.579232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:12.580153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:12.581463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:12.581463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:12.581463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:12.581463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 249/499] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:12.581463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5277c9ad-bbe4-46d8-9429-c9c998e8b83a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.610718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d51877-f290-4979-ab12-bb01965a1224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.611304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d51877-f290-4979-ab12-bb01965a1224 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.611586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d51877-f290-4979-ab12-bb01965a1224 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.611851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d51877-f290-4979-ab12-bb01965a1224 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.612537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d51877-f290-4979-ab12-bb01965a1224 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:12.612861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d51877-f290-4979-ab12-bb01965a1224 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:12.614243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d51877-f290-4979-ab12-bb01965a1224 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:12.616320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:12.625034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 251/500] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:12.625831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d51877-f290-4979-ab12-bb01965a1224 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.686458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5277c9ad-bbe4-46d8-9429-c9c998e8b83a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.719569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb84f1c5-7fc3-4f9f-a405-f35352c7bb66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.720191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb84f1c5-7fc3-4f9f-a405-f35352c7bb66 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.720512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb84f1c5-7fc3-4f9f-a405-f35352c7bb66 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.720794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb84f1c5-7fc3-4f9f-a405-f35352c7bb66 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.724311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d51877-f290-4979-ab12-bb01965a1224 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.731214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb84f1c5-7fc3-4f9f-a405-f35352c7bb66 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:12.732708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb84f1c5-7fc3-4f9f-a405-f35352c7bb66 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-fb84f1c5-7fc3-4f9f-a405-f35352c7bb66 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:12.747714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:12.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:12.749553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:12.749553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:12.749553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:12.749553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 250/501] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:12.750369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18b90b1-25d1-4551-85cb-fab544bb2fa8 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.750644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18b90b1-25d1-4551-85cb-fab544bb2fa8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.750908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18b90b1-25d1-4551-85cb-fab544bb2fa8 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.752105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 251/502] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:12 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:12.752879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18b90b1-25d1-4551-85cb-fab544bb2fa8 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.760877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:12.763300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5730930b-02c5-469b-87d3-1d272a0d84f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.763947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5730930b-02c5-469b-87d3-1d272a0d84f5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.764230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5730930b-02c5-469b-87d3-1d272a0d84f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.764546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5730930b-02c5-469b-87d3-1d272a0d84f5 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.765281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5730930b-02c5-469b-87d3-1d272a0d84f5 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:12.765595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5730930b-02c5-469b-87d3-1d272a0d84f5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:12.767039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5730930b-02c5-469b-87d3-1d272a0d84f5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:12.803539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 252/503] 10.222.0.25 () {56 vars in 1133 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:12.804168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5730930b-02c5-469b-87d3-1d272a0d84f5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.860484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:12.864371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18b90b1-25d1-4551-85cb-fab544bb2fa8 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.898141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9051f422-4b47-4bad-963e-3e6266f02eb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.898757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9051f422-4b47-4bad-963e-3e6266f02eb1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.899014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9051f422-4b47-4bad-963e-3e6266f02eb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.899374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9051f422-4b47-4bad-963e-3e6266f02eb1 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.900022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9051f422-4b47-4bad-963e-3e6266f02eb1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:12.900316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9051f422-4b47-4bad-963e-3e6266f02eb1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:12.901692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9051f422-4b47-4bad-963e-3e6266f02eb1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:12.903257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5730930b-02c5-469b-87d3-1d272a0d84f5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:12.912774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 252/504] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:12.913964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a23dce1-3100-453f-8aed-53ff372d1615 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.914243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a23dce1-3100-453f-8aed-53ff372d1615 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.914516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a23dce1-3100-453f-8aed-53ff372d1615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.933490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-3a23dce1-3100-453f-8aed-53ff372d1615 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:12.934739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3a23dce1-3100-453f-8aed-53ff372d1615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:12.937896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d480ef15-4795-4c8d-a688-6b7a465349f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:12.938471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d480ef15-4795-4c8d-a688-6b7a465349f2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:12.938751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d480ef15-4795-4c8d-a688-6b7a465349f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:12.939024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d480ef15-4795-4c8d-a688-6b7a465349f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:12.949423 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d480ef15-4795-4c8d-a688-6b7a465349f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:12.950885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d480ef15-4795-4c8d-a688-6b7a465349f2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d480ef15-4795-4c8d-a688-6b7a465349f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:12.965913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:12.967007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:12.968247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:12.968247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:12.968247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:12.968247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 253/505] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:12.968247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d480ef15-4795-4c8d-a688-6b7a465349f2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:12.996014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:13.004363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 253/506] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:13.005084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a23dce1-3100-453f-8aed-53ff372d1615 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.071446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d480ef15-4795-4c8d-a688-6b7a465349f2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:13.083736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:13.111335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7b8090-4451-4ffe-b84c-b4b13f31862a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:13.112024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7b8090-4451-4ffe-b84c-b4b13f31862a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:13.112306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7b8090-4451-4ffe-b84c-b4b13f31862a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:13.112825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7b8090-4451-4ffe-b84c-b4b13f31862a None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:13.113238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7b8090-4451-4ffe-b84c-b4b13f31862a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:13.113541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7b8090-4451-4ffe-b84c-b4b13f31862a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:13.114918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7b8090-4451-4ffe-b84c-b4b13f31862a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:13.128718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 254/507] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:13.130074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7b8090-4451-4ffe-b84c-b4b13f31862a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.138605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a23dce1-3100-453f-8aed-53ff372d1615 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:13.173776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a6c47d-31e2-4b1c-8922-f576367dbf66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:13.174376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a6c47d-31e2-4b1c-8922-f576367dbf66 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:13.174754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a6c47d-31e2-4b1c-8922-f576367dbf66 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:13.174926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a6c47d-31e2-4b1c-8922-f576367dbf66 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:13.237269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:13.246496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a6c47d-31e2-4b1c-8922-f576367dbf66 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:13.248477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a6c47d-31e2-4b1c-8922-f576367dbf66 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:13.250734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7b8090-4451-4ffe-b84c-b4b13f31862a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:13.287014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94818286-a784-497a-b411-ec67ad64f3d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:13.287632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94818286-a784-497a-b411-ec67ad64f3d0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:13.287910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94818286-a784-497a-b411-ec67ad64f3d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:13.288178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94818286-a784-497a-b411-ec67ad64f3d0 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:13.295477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94818286-a784-497a-b411-ec67ad64f3d0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:13.296969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94818286-a784-497a-b411-ec67ad64f3d0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-94818286-a784-497a-b411-ec67ad64f3d0 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:13.308978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:13.309978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 255/508] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/users/glance => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:13.310884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94818286-a784-497a-b411-ec67ad64f3d0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.338550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:13.354183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 254/509] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:13.355013 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a6c47d-31e2-4b1c-8922-f576367dbf66 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.412501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94818286-a784-497a-b411-ec67ad64f3d0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:13.445666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:13.445964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb1e4f-ee99-4bfe-a0c9-4d79efdded3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:13.446651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb1e4f-ee99-4bfe-a0c9-4d79efdded3c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:13.446932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb1e4f-ee99-4bfe-a0c9-4d79efdded3c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:13.447212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb1e4f-ee99-4bfe-a0c9-4d79efdded3c None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:13.448146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb1e4f-ee99-4bfe-a0c9-4d79efdded3c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:13.448496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb1e4f-ee99-4bfe-a0c9-4d79efdded3c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:13.449877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb1e4f-ee99-4bfe-a0c9-4d79efdded3c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:13.453426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a6c47d-31e2-4b1c-8922-f576367dbf66 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:13.462263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 256/510] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:13.463247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb1e4f-ee99-4bfe-a0c9-4d79efdded3c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.492096 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645bf6b9-bf3d-4e00-8600-30471204fb54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:13.492678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645bf6b9-bf3d-4e00-8600-30471204fb54 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:13.492961 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645bf6b9-bf3d-4e00-8600-30471204fb54 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:13.493315 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645bf6b9-bf3d-4e00-8600-30471204fb54 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:13.500256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645bf6b9-bf3d-4e00-8600-30471204fb54 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:13.501708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645bf6b9-bf3d-4e00-8600-30471204fb54 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-645bf6b9-bf3d-4e00-8600-30471204fb54 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:13.515044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:13.515927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:13.516876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 255/511] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/users/cinder => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:13.517441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645bf6b9-bf3d-4e00-8600-30471204fb54 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.552852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:13.565955 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb1e4f-ee99-4bfe-a0c9-4d79efdded3c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:13.600265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea107c22-4875-42ae-aeeb-b21fc4ff0b01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:13.600908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea107c22-4875-42ae-aeeb-b21fc4ff0b01 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:13.601187 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea107c22-4875-42ae-aeeb-b21fc4ff0b01 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:13.601462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea107c22-4875-42ae-aeeb-b21fc4ff0b01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:13.611665 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea107c22-4875-42ae-aeeb-b21fc4ff0b01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:13.613147 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea107c22-4875-42ae-aeeb-b21fc4ff0b01 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:13.617280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645bf6b9-bf3d-4e00-8600-30471204fb54 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ea107c22-4875-42ae-aeeb-b21fc4ff0b01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:13.628381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:13.629477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:13.630731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:13.630731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:13.630731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:13.630731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 257/512] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:13.630731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea107c22-4875-42ae-aeeb-b21fc4ff0b01 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.654173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fad6c36-d6b3-4643-b34f-3682394d960d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:13.654770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fad6c36-d6b3-4643-b34f-3682394d960d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:13.655046 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fad6c36-d6b3-4643-b34f-3682394d960d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:13.655317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fad6c36-d6b3-4643-b34f-3682394d960d None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:13.656049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fad6c36-d6b3-4643-b34f-3682394d960d None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:13.656362 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fad6c36-d6b3-4643-b34f-3682394d960d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:13.657780 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fad6c36-d6b3-4643-b34f-3682394d960d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:13.695503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 256/513] 10.222.0.25 () {56 vars in 1137 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:13.696895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fad6c36-d6b3-4643-b34f-3682394d960d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.713310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:13.729759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea107c22-4875-42ae-aeeb-b21fc4ff0b01 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:13.763076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a300684f-081c-4481-b0a3-1beb7a012030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:13.763624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a300684f-081c-4481-b0a3-1beb7a012030 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:13.763902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a300684f-081c-4481-b0a3-1beb7a012030 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:13.764208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a300684f-081c-4481-b0a3-1beb7a012030 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:13.773339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:13.776071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a300684f-081c-4481-b0a3-1beb7a012030 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:13.777771 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a300684f-081c-4481-b0a3-1beb7a012030 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a300684f-081c-4481-b0a3-1beb7a012030 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:13.792185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:13.793142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:13.794040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:13.794040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:13.794040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:13.794040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 258/514] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:13.794040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a300684f-081c-4481-b0a3-1beb7a012030 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.801646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fad6c36-d6b3-4643-b34f-3682394d960d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:13.835463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d824cf71-8ad7-4d35-a0fb-a33f679714a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:13.836087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d824cf71-8ad7-4d35-a0fb-a33f679714a2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:13.836391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d824cf71-8ad7-4d35-a0fb-a33f679714a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:13.836667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d824cf71-8ad7-4d35-a0fb-a33f679714a2 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:13.843593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d824cf71-8ad7-4d35-a0fb-a33f679714a2 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:13.845082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d824cf71-8ad7-4d35-a0fb-a33f679714a2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-d824cf71-8ad7-4d35-a0fb-a33f679714a2 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:13.856934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:13.857843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:13.858731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 257/515] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/users/neutron => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:13.859499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d824cf71-8ad7-4d35-a0fb-a33f679714a2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.894583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a300684f-081c-4481-b0a3-1beb7a012030 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:13.926778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7936c935-0f1f-463b-b349-330015dabd04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:13.927372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7936c935-0f1f-463b-b349-330015dabd04 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:13.927652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7936c935-0f1f-463b-b349-330015dabd04 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:13.927927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7936c935-0f1f-463b-b349-330015dabd04 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:13.928681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7936c935-0f1f-463b-b349-330015dabd04 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:13.928988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7936c935-0f1f-463b-b349-330015dabd04 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:13.930427 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7936c935-0f1f-463b-b349-330015dabd04 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:13.934328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:13.950186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 259/516] 10.222.0.25 () {56 vars in 1137 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:13.951024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7936c935-0f1f-463b-b349-330015dabd04 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:13.967766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d824cf71-8ad7-4d35-a0fb-a33f679714a2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:14.000407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:14.002881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3325457a-a6df-4ead-b293-1466a18d73ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:14.003517 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3325457a-a6df-4ead-b293-1466a18d73ae None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:14.003771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3325457a-a6df-4ead-b293-1466a18d73ae None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:14.004073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3325457a-a6df-4ead-b293-1466a18d73ae None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:14.004749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3325457a-a6df-4ead-b293-1466a18d73ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:14.005061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3325457a-a6df-4ead-b293-1466a18d73ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:14.006430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3325457a-a6df-4ead-b293-1466a18d73ae None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:14.018625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 258/517] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:14.019373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3325457a-a6df-4ead-b293-1466a18d73ae None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:14.051767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7936c935-0f1f-463b-b349-330015dabd04 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:14.086434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e52d78-4aef-4873-861a-4ca0b466b6a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:14.087269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e52d78-4aef-4873-861a-4ca0b466b6a9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:14.087722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e52d78-4aef-4873-861a-4ca0b466b6a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:14.087999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e52d78-4aef-4873-861a-4ca0b466b6a9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:14.123482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3325457a-a6df-4ead-b293-1466a18d73ae None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:14.152035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:14.154799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e52d78-4aef-4873-861a-4ca0b466b6a9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:14.156268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e52d78-4aef-4873-861a-4ca0b466b6a9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:14.157392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fff13b1-7ede-445e-8563-3856f2fa57b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:14.157992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff13b1-7ede-445e-8563-3856f2fa57b5 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:14.158273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff13b1-7ede-445e-8563-3856f2fa57b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:14.158539 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff13b1-7ede-445e-8563-3856f2fa57b5 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:14.159176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fff13b1-7ede-445e-8563-3856f2fa57b5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:14.159475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fff13b1-7ede-445e-8563-3856f2fa57b5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:14.160871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fff13b1-7ede-445e-8563-3856f2fa57b5 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:14.172956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 259/518] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:14.173870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fff13b1-7ede-445e-8563-3856f2fa57b5 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:14.204412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:14.255448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 260/519] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:14.256279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e52d78-4aef-4873-861a-4ca0b466b6a9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:14.273254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fff13b1-7ede-445e-8563-3856f2fa57b5 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:14.306121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b1c638-c8a3-4b82-805b-e086f3ceee91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:14.306708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b1c638-c8a3-4b82-805b-e086f3ceee91 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:14.306982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b1c638-c8a3-4b82-805b-e086f3ceee91 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:14.307248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b1c638-c8a3-4b82-805b-e086f3ceee91 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:14.307966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b1c638-c8a3-4b82-805b-e086f3ceee91 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:14.308264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b1c638-c8a3-4b82-805b-e086f3ceee91 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:14.309701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b1c638-c8a3-4b82-805b-e086f3ceee91 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:14.327116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 260/520] 10.222.0.25 () {56 vars in 1139 bytes} [Mon Sep 26 01:23:14 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:14.327898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b1c638-c8a3-4b82-805b-e086f3ceee91 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:14.358674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e52d78-4aef-4873-861a-4ca0b466b6a9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:14.377842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:14.394214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7cddac-5bcd-4b29-8c8a-4c8b39614331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:14.394809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7cddac-5bcd-4b29-8c8a-4c8b39614331 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:14.395079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7cddac-5bcd-4b29-8c8a-4c8b39614331 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:14.395349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7cddac-5bcd-4b29-8c8a-4c8b39614331 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:14.414994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:14.428456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b1c638-c8a3-4b82-805b-e086f3ceee91 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:14.463314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d3d542-9d90-48b6-afe0-fea2f9390658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:14.464023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d3d542-9d90-48b6-afe0-fea2f9390658 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:14.464299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d3d542-9d90-48b6-afe0-fea2f9390658 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:14.464611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d3d542-9d90-48b6-afe0-fea2f9390658 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:14.465227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d3d542-9d90-48b6-afe0-fea2f9390658 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:14.466181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7cddac-5bcd-4b29-8c8a-4c8b39614331 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:14.466657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d3d542-9d90-48b6-afe0-fea2f9390658 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:14.467656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7cddac-5bcd-4b29-8c8a-4c8b39614331 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:14.487511 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 261/521] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:23:14 2022] POST /identity/v3/users => generated 247 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:14.488321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d3d542-9d90-48b6-afe0-fea2f9390658 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:14.597984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 261/522] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:14.599176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7cddac-5bcd-4b29-8c8a-4c8b39614331 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:14.614131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d3d542-9d90-48b6-afe0-fea2f9390658 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:14.623538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:14.655920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4d70b1-7d9c-4e46-9382-9d19c67da442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:14.656585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4d70b1-7d9c-4e46-9382-9d19c67da442 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:14.656865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4d70b1-7d9c-4e46-9382-9d19c67da442 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:14.657139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4d70b1-7d9c-4e46-9382-9d19c67da442 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:14.659445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:14.666018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4d70b1-7d9c-4e46-9382-9d19c67da442 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:14.667446 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4d70b1-7d9c-4e46-9382-9d19c67da442 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ee4d70b1-7d9c-4e46-9382-9d19c67da442 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:14.682636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:14.683571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:14.685228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:14.685228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:14.685228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:14.685228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:14.685228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:14.685228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:14.685228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:14.685228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:14.685228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 262/523] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:14.685228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4d70b1-7d9c-4e46-9382-9d19c67da442 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:14.725219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7cddac-5bcd-4b29-8c8a-4c8b39614331 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:14.762551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c331193-7e7c-4fe3-a4ca-417b829cca7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:14.763280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c331193-7e7c-4fe3-a4ca-417b829cca7c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:14.763577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c331193-7e7c-4fe3-a4ca-417b829cca7c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:14.763854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c331193-7e7c-4fe3-a4ca-417b829cca7c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:14.773945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c331193-7e7c-4fe3-a4ca-417b829cca7c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:14.775647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c331193-7e7c-4fe3-a4ca-417b829cca7c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-9c331193-7e7c-4fe3-a4ca-417b829cca7c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:14.790638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:14.791585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:14.792480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:14.792480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:14.792480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:14.792480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 262/524] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:14.793181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c331193-7e7c-4fe3-a4ca-417b829cca7c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:14.804256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4d70b1-7d9c-4e46-9382-9d19c67da442 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:14.840921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e24298-d9fe-49d7-911b-fc8e4fa46191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:14.841578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e24298-d9fe-49d7-911b-fc8e4fa46191 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:14.841795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e24298-d9fe-49d7-911b-fc8e4fa46191 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:14.842070 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e24298-d9fe-49d7-911b-fc8e4fa46191 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:14.882270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:14.905413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c331193-7e7c-4fe3-a4ca-417b829cca7c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:14.907175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:14.924941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e24298-d9fe-49d7-911b-fc8e4fa46191 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:14.926389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e24298-d9fe-49d7-911b-fc8e4fa46191 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:14.950128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851b74e7-ee88-4eed-b736-9d74506cecb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:14.950464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851b74e7-ee88-4eed-b736-9d74506cecb4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:14.950709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851b74e7-ee88-4eed-b736-9d74506cecb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:14.950970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851b74e7-ee88-4eed-b736-9d74506cecb4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:14.964619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851b74e7-ee88-4eed-b736-9d74506cecb4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:14.966220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851b74e7-ee88-4eed-b736-9d74506cecb4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-851b74e7-ee88-4eed-b736-9d74506cecb4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:14.982267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:14.983279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:14.984136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:14.984136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:14.984136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:14.984136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 263/525] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:14.984551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851b74e7-ee88-4eed-b736-9d74506cecb4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:15.050355 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 263/526] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:15.052384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e24298-d9fe-49d7-911b-fc8e4fa46191 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:15.111101 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851b74e7-ee88-4eed-b736-9d74506cecb4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:15.137121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:15.153340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2270cff8-d407-4b07-9653-7ba6886b0842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:15.154702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2270cff8-d407-4b07-9653-7ba6886b0842 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:15.154990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2270cff8-d407-4b07-9653-7ba6886b0842 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:15.155397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2270cff8-d407-4b07-9653-7ba6886b0842 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:15.156823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2270cff8-d407-4b07-9653-7ba6886b0842 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:15.157137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2270cff8-d407-4b07-9653-7ba6886b0842 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:15.158603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2270cff8-d407-4b07-9653-7ba6886b0842 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:15.182099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 264/527] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:15.183190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e24298-d9fe-49d7-911b-fc8e4fa46191 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:15.184221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2270cff8-d407-4b07-9653-7ba6886b0842 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:15.203582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:15.233982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d857b9-84bb-4973-8fca-95c405badc5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:15.235117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d857b9-84bb-4973-8fca-95c405badc5b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:15.235795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d857b9-84bb-4973-8fca-95c405badc5b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:15.236298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d857b9-84bb-4973-8fca-95c405badc5b None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:15.237529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d857b9-84bb-4973-8fca-95c405badc5b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:15.238067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d857b9-84bb-4973-8fca-95c405badc5b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:15.239576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d857b9-84bb-4973-8fca-95c405badc5b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:15.260324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 264/528] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:15.260324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d857b9-84bb-4973-8fca-95c405badc5b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:15.304484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2270cff8-d407-4b07-9653-7ba6886b0842 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:15.344571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ef05d0-f9f9-4933-a9e5-4e7dac7545b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:15.345177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ef05d0-f9f9-4933-a9e5-4e7dac7545b5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:15.345454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ef05d0-f9f9-4933-a9e5-4e7dac7545b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:15.345849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ef05d0-f9f9-4933-a9e5-4e7dac7545b5 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:15.346504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ef05d0-f9f9-4933-a9e5-4e7dac7545b5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:15.346804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ef05d0-f9f9-4933-a9e5-4e7dac7545b5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:15.348272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ef05d0-f9f9-4933-a9e5-4e7dac7545b5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:15.367102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 265/529] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:15.367102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ef05d0-f9f9-4933-a9e5-4e7dac7545b5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:15.392522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d857b9-84bb-4973-8fca-95c405badc5b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:15.422001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:15.432829 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bab3ae-777b-4e96-ba96-6542846a99e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:15.433487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bab3ae-777b-4e96-ba96-6542846a99e1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:15.433768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bab3ae-777b-4e96-ba96-6542846a99e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:15.434071 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bab3ae-777b-4e96-ba96-6542846a99e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:15.447305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bab3ae-777b-4e96-ba96-6542846a99e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:15.449005 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bab3ae-777b-4e96-ba96-6542846a99e1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-70bab3ae-777b-4e96-ba96-6542846a99e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:15.468466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:15.469496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:15.470434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:15.470434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:15.470434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:15.470661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:15.475239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 265/530] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:15.476135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bab3ae-777b-4e96-ba96-6542846a99e1 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:15.499825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ef05d0-f9f9-4933-a9e5-4e7dac7545b5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:15.537393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b0201c-e4d4-4ba5-9b92-8d96cf33d6c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:15.538409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b0201c-e4d4-4ba5-9b92-8d96cf33d6c5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:15.538861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b0201c-e4d4-4ba5-9b92-8d96cf33d6c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:15.539379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b0201c-e4d4-4ba5-9b92-8d96cf33d6c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:15.541641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b0201c-e4d4-4ba5-9b92-8d96cf33d6c5 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=381ad7c239f444da90369dc58779b86c) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:15.542586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b0201c-e4d4-4ba5-9b92-8d96cf33d6c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:15.544035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b0201c-e4d4-4ba5-9b92-8d96cf33d6c5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:15.560376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 266/531] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:15 2022] GET /identity/v3/role_assignments?user.id=381ad7c239f444da90369dc58779b86c&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:15.560376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b0201c-e4d4-4ba5-9b92-8d96cf33d6c5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:15.620709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bab3ae-777b-4e96-ba96-6542846a99e1 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:15.662800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e35bf0-c47b-429a-8752-7d1c976896df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:15.663875 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e35bf0-c47b-429a-8752-7d1c976896df None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:15.664207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e35bf0-c47b-429a-8752-7d1c976896df None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:15.664544 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e35bf0-c47b-429a-8752-7d1c976896df None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:15.674307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e35bf0-c47b-429a-8752-7d1c976896df None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:15.675979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e35bf0-c47b-429a-8752-7d1c976896df None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-e1e35bf0-c47b-429a-8752-7d1c976896df None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:15.689737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:15.690655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:15.691506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:15.691506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:15.691506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:15.691506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:15.691506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:15.691506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:15.691506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:15.691506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:15.696793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 266/532] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:15.698045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e35bf0-c47b-429a-8752-7d1c976896df None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:15.701154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b0201c-e4d4-4ba5-9b92-8d96cf33d6c5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:15.747929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:15.750046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b60cf96-0653-4c35-a3e2-3fcaaaccb0f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:15.750753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b60cf96-0653-4c35-a3e2-3fcaaaccb0f6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:15.751201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b60cf96-0653-4c35-a3e2-3fcaaaccb0f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:15.751575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b60cf96-0653-4c35-a3e2-3fcaaaccb0f6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:15.761555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b60cf96-0653-4c35-a3e2-3fcaaaccb0f6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:15.763165 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b60cf96-0653-4c35-a3e2-3fcaaaccb0f6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:15.765752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-7b60cf96-0653-4c35-a3e2-3fcaaaccb0f6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:15.782500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:15.783516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:15.784420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:15.784420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:15.784420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:15.784420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:15.784420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:15.784420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:15.784420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:15.784420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:15.787656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 267/533] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:15.789797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b60cf96-0653-4c35-a3e2-3fcaaaccb0f6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:15.825465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e35bf0-c47b-429a-8752-7d1c976896df None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:15.875717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a8fe5a-9c34-4c9b-bbb6-1ad931203507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:15.876513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a8fe5a-9c34-4c9b-bbb6-1ad931203507 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:15.876926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a8fe5a-9c34-4c9b-bbb6-1ad931203507 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:15.877201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a8fe5a-9c34-4c9b-bbb6-1ad931203507 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:15.877864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a8fe5a-9c34-4c9b-bbb6-1ad931203507 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:15.878167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a8fe5a-9c34-4c9b-bbb6-1ad931203507 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:15.879550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a8fe5a-9c34-4c9b-bbb6-1ad931203507 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:15.899871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 267/534] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:15.901883 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a8fe5a-9c34-4c9b-bbb6-1ad931203507 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:15.910746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b60cf96-0653-4c35-a3e2-3fcaaaccb0f6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:15.952919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48053dd8-7f60-49cc-b55e-39f7a20681a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:15.953525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48053dd8-7f60-49cc-b55e-39f7a20681a2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:15.953804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48053dd8-7f60-49cc-b55e-39f7a20681a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:15.954078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48053dd8-7f60-49cc-b55e-39f7a20681a2 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:15.954876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48053dd8-7f60-49cc-b55e-39f7a20681a2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:15.955190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48053dd8-7f60-49cc-b55e-39f7a20681a2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:15.957034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48053dd8-7f60-49cc-b55e-39f7a20681a2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:15.971128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 268/535] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:15.972128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48053dd8-7f60-49cc-b55e-39f7a20681a2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:16.011681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:16.015423 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a8fe5a-9c34-4c9b-bbb6-1ad931203507 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:16.037534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:16.058183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e1ffc3-28a4-470e-91e2-6921c84c83da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:16.058804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e1ffc3-28a4-470e-91e2-6921c84c83da None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:16.059085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e1ffc3-28a4-470e-91e2-6921c84c83da None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:16.059357 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e1ffc3-28a4-470e-91e2-6921c84c83da None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:16.060305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e1ffc3-28a4-470e-91e2-6921c84c83da None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:16.060593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e1ffc3-28a4-470e-91e2-6921c84c83da None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:16.062251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e1ffc3-28a4-470e-91e2-6921c84c83da None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:16.085538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48053dd8-7f60-49cc-b55e-39f7a20681a2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:16.087451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 268/536] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:16.089211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e1ffc3-28a4-470e-91e2-6921c84c83da None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:16.125262 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac9b281-7007-4172-8da9-bdf64cc132ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:16.126545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac9b281-7007-4172-8da9-bdf64cc132ae None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:16.127217 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac9b281-7007-4172-8da9-bdf64cc132ae None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:16.127637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac9b281-7007-4172-8da9-bdf64cc132ae None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:16.137738 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac9b281-7007-4172-8da9-bdf64cc132ae None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:16.139407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac9b281-7007-4172-8da9-bdf64cc132ae None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-eac9b281-7007-4172-8da9-bdf64cc132ae None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:16.157121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:16.158218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:16.159246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:16.159246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:16.159246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:16.159246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:16.159246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:16.159246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:16.159246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:16.159246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:16.161866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 269/537] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:16.162806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac9b281-7007-4172-8da9-bdf64cc132ae None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:16.219542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e1ffc3-28a4-470e-91e2-6921c84c83da None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:16.267296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a063d53a-466e-4053-b9e7-b095b36bcd21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:16.268254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a063d53a-466e-4053-b9e7-b095b36bcd21 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:16.268700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a063d53a-466e-4053-b9e7-b095b36bcd21 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:16.269263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a063d53a-466e-4053-b9e7-b095b36bcd21 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:16.270170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a063d53a-466e-4053-b9e7-b095b36bcd21 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=e07b6890d3004dd78faed48a1ab90086) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:16.270711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a063d53a-466e-4053-b9e7-b095b36bcd21 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:16.273059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a063d53a-466e-4053-b9e7-b095b36bcd21 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:16.280031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 269/538] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:16 2022] GET /identity/v3/role_assignments?user.id=e07b6890d3004dd78faed48a1ab90086&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 259 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:16.281077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a063d53a-466e-4053-b9e7-b095b36bcd21 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:16.289083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac9b281-7007-4172-8da9-bdf64cc132ae None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:16.309619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:16.319015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:16.332935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552f5734-8240-4b3b-86cf-852c4c72499c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:16.334312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552f5734-8240-4b3b-86cf-852c4c72499c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:16.334312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552f5734-8240-4b3b-86cf-852c4c72499c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:16.334312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552f5734-8240-4b3b-86cf-852c4c72499c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:16.335117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552f5734-8240-4b3b-86cf-852c4c72499c None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=2bb5f76eb08741f3a8c7a3a140e01e24) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:16.335495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552f5734-8240-4b3b-86cf-852c4c72499c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:16.337172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552f5734-8240-4b3b-86cf-852c4c72499c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:16.344232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 270/539] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:16 2022] GET /identity/v3/role_assignments?user.id=2bb5f76eb08741f3a8c7a3a140e01e24&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:16.420873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a063d53a-466e-4053-b9e7-b095b36bcd21 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:16.462840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6776a60-88d4-4f71-8d33-7deb4e933ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:16.463773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6776a60-88d4-4f71-8d33-7deb4e933ca4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:16.464231 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6776a60-88d4-4f71-8d33-7deb4e933ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:16.464702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6776a60-88d4-4f71-8d33-7deb4e933ca4 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:16.465618 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6776a60-88d4-4f71-8d33-7deb4e933ca4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:16.466102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6776a60-88d4-4f71-8d33-7deb4e933ca4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:16.467735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6776a60-88d4-4f71-8d33-7deb4e933ca4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:16.482257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 270/540] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:16.488120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552f5734-8240-4b3b-86cf-852c4c72499c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:16.629203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552f5734-8240-4b3b-86cf-852c4c72499c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:16.681364 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453616a4-b86b-40e3-a133-fee5cbc3b087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:16.682340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453616a4-b86b-40e3-a133-fee5cbc3b087 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:16.682971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453616a4-b86b-40e3-a133-fee5cbc3b087 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:16.683443 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453616a4-b86b-40e3-a133-fee5cbc3b087 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:16.684414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453616a4-b86b-40e3-a133-fee5cbc3b087 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=0478ebab37fb4f488eef84f7a241e206) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:16.684979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453616a4-b86b-40e3-a133-fee5cbc3b087 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:16.686693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453616a4-b86b-40e3-a133-fee5cbc3b087 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:16.692887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 271/541] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:16 2022] GET /identity/v3/role_assignments?user.id=0478ebab37fb4f488eef84f7a241e206&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 259 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:17.029406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3667e1b9-f62e-4377-82c3-1b813ecac02f None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:17.029940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3667e1b9-f62e-4377-82c3-1b813ecac02f None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:17.029940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3667e1b9-f62e-4377-82c3-1b813ecac02f None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:17.031561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 271/542] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:17.044878 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020738f6-187d-4016-b95a-73e77c6acecb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:17.045376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020738f6-187d-4016-b95a-73e77c6acecb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:17.045465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020738f6-187d-4016-b95a-73e77c6acecb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:17.064825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-020738f6-187d-4016-b95a-73e77c6acecb None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:17.066023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-020738f6-187d-4016-b95a-73e77c6acecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:17.126914 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:17.160907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 272/543] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:23:17.544768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3667e1b9-f62e-4377-82c3-1b813ecac02f None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:17.677863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:17.708286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3667e1b9-f62e-4377-82c3-1b813ecac02f None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:17.756796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c8bbd-d3d0-4833-8274-7fa8d6824ec6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:17.757420 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c8bbd-d3d0-4833-8274-7fa8d6824ec6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:17.757695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c8bbd-d3d0-4833-8274-7fa8d6824ec6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:17.757966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816c8bbd-d3d0-4833-8274-7fa8d6824ec6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:17.778835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816c8bbd-d3d0-4833-8274-7fa8d6824ec6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-816c8bbd-d3d0-4833-8274-7fa8d6824ec6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:17.783692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:17.784657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:17.785455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 272/544] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:17.791397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020738f6-187d-4016-b95a-73e77c6acecb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:17.906210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020738f6-187d-4016-b95a-73e77c6acecb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:17.951785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2849ab3-544f-40c6-9c3e-089be5f1734c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:17.952474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2849ab3-544f-40c6-9c3e-089be5f1734c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:17.952755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2849ab3-544f-40c6-9c3e-089be5f1734c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:17.953026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2849ab3-544f-40c6-9c3e-089be5f1734c None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:17.953685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2849ab3-544f-40c6-9c3e-089be5f1734c None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:17.953985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2849ab3-544f-40c6-9c3e-089be5f1734c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:17.955355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2849ab3-544f-40c6-9c3e-089be5f1734c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:17.968179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 273/545] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:17.973893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c8bbd-d3d0-4833-8274-7fa8d6824ec6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:18.129050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7fc98-db66-45d3-87fc-7c288ceb1196 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:18.129560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7fc98-db66-45d3-87fc-7c288ceb1196 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:18.129560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7fc98-db66-45d3-87fc-7c288ceb1196 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:18.131073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816c8bbd-d3d0-4833-8274-7fa8d6824ec6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:18.131449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 274/546] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:23:18.145616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e61a1cb-7a9f-4d96-937a-00d6ed7d6234 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:18.145905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e61a1cb-7a9f-4d96-937a-00d6ed7d6234 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:18.146182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e61a1cb-7a9f-4d96-937a-00d6ed7d6234 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:18.176523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64216d38-8f4e-45cf-8749-e95091272d00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:18.177490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64216d38-8f4e-45cf-8749-e95091272d00 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:18.177965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64216d38-8f4e-45cf-8749-e95091272d00 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:18.178453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64216d38-8f4e-45cf-8749-e95091272d00 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:18.190636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2e61a1cb-7a9f-4d96-937a-00d6ed7d6234 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:18.191885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2e61a1cb-7a9f-4d96-937a-00d6ed7d6234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:18.204473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64216d38-8f4e-45cf-8749-e95091272d00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:18.206539 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64216d38-8f4e-45cf-8749-e95091272d00 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:18.212648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:18.225953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-64216d38-8f4e-45cf-8749-e95091272d00 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:18.236810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:18.237721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:18.238593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:18.238593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:18.238593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:18.238593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 273/547] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:18.243507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64216d38-8f4e-45cf-8749-e95091272d00 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:18.301104 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 275/548] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:18.406439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64216d38-8f4e-45cf-8749-e95091272d00 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:18.450323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6351cb-4e85-499b-b1e9-09132d960dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:18.450933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6351cb-4e85-499b-b1e9-09132d960dd3 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:18.451207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6351cb-4e85-499b-b1e9-09132d960dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:18.451476 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6351cb-4e85-499b-b1e9-09132d960dd3 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:18.452115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6351cb-4e85-499b-b1e9-09132d960dd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:18.452445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6351cb-4e85-499b-b1e9-09132d960dd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:18.453797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6351cb-4e85-499b-b1e9-09132d960dd3 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:18.470116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 274/549] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:18.475391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e61a1cb-7a9f-4d96-937a-00d6ed7d6234 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:18.610358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6351cb-4e85-499b-b1e9-09132d960dd3 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:18.614225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e61a1cb-7a9f-4d96-937a-00d6ed7d6234 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:18.689096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864cb9ff-737b-47a7-9548-576364241b4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:18.689733 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864cb9ff-737b-47a7-9548-576364241b4f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:18.690002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864cb9ff-737b-47a7-9548-576364241b4f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:18.690267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864cb9ff-737b-47a7-9548-576364241b4f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:18.704733 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864cb9ff-737b-47a7-9548-576364241b4f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:18.706138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864cb9ff-737b-47a7-9548-576364241b4f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:18.707110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-864cb9ff-737b-47a7-9548-576364241b4f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:18.729102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:18.730317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:18.731237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:18.731237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:18.731237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:18.731237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:18.731237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:18.731237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:18.731237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:18.731237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:18.731237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:18.731237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:18.732240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 276/550] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:18.737795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864cb9ff-737b-47a7-9548-576364241b4f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:18.787525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6351cb-4e85-499b-b1e9-09132d960dd3 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:18.787951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:18.855790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e8fb56-c8a7-4a81-9972-9ee5597903ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:18.856524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e8fb56-c8a7-4a81-9972-9ee5597903ef None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:18.856803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e8fb56-c8a7-4a81-9972-9ee5597903ef None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:18.857073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e8fb56-c8a7-4a81-9972-9ee5597903ef None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:18.879883 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e8fb56-c8a7-4a81-9972-9ee5597903ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-76e8fb56-c8a7-4a81-9972-9ee5597903ef None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:18.887263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:18.888159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:18.889214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:18.889856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 275/551] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:18.898218 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e8fb56-c8a7-4a81-9972-9ee5597903ef None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:18.925315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864cb9ff-737b-47a7-9548-576364241b4f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:18.985006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c14cc2-009c-419f-84f1-6ec2e7298f41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:18.986061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c14cc2-009c-419f-84f1-6ec2e7298f41 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:18.986539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c14cc2-009c-419f-84f1-6ec2e7298f41 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:18.987026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c14cc2-009c-419f-84f1-6ec2e7298f41 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:18.987956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c14cc2-009c-419f-84f1-6ec2e7298f41 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:18.988583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c14cc2-009c-419f-84f1-6ec2e7298f41 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:18.990323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c14cc2-009c-419f-84f1-6ec2e7298f41 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:19.011068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e8fb56-c8a7-4a81-9972-9ee5597903ef None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:19.028279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:19.034270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 277/552] 10.222.0.25 () {56 vars in 1135 bytes} [Mon Sep 26 01:23:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:19.042996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c14cc2-009c-419f-84f1-6ec2e7298f41 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:19.060737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092d5449-e03c-4b27-b31b-9d7aba47193a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:19.061348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092d5449-e03c-4b27-b31b-9d7aba47193a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.061623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092d5449-e03c-4b27-b31b-9d7aba47193a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.061896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092d5449-e03c-4b27-b31b-9d7aba47193a None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.062568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092d5449-e03c-4b27-b31b-9d7aba47193a None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:19.062868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092d5449-e03c-4b27-b31b-9d7aba47193a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:19.065229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-092d5449-e03c-4b27-b31b-9d7aba47193a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:19.083067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 276/553] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:19.090114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092d5449-e03c-4b27-b31b-9d7aba47193a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:19.192410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:19.209525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c14cc2-009c-419f-84f1-6ec2e7298f41 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:19.209843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092d5449-e03c-4b27-b31b-9d7aba47193a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:19.247724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c13c857-8715-4dce-9a1a-0d9f6cc1e71a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:19.248149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef65d7a4-9063-48ad-a058-b6204bce2797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:19.248267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c13c857-8715-4dce-9a1a-0d9f6cc1e71a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.248544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c13c857-8715-4dce-9a1a-0d9f6cc1e71a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.248806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef65d7a4-9063-48ad-a058-b6204bce2797 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.248916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c13c857-8715-4dce-9a1a-0d9f6cc1e71a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.249084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef65d7a4-9063-48ad-a058-b6204bce2797 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.249410 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef65d7a4-9063-48ad-a058-b6204bce2797 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.260579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef65d7a4-9063-48ad-a058-b6204bce2797 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:19.262432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef65d7a4-9063-48ad-a058-b6204bce2797 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ef65d7a4-9063-48ad-a058-b6204bce2797 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:19.277801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:19.278716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:19.279589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:19.279589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:19.279589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:19.279589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 277/554] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:19.280551 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8483d-1de7-4272-ab23-127c01e065f8 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.280832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8483d-1de7-4272-ab23-127c01e065f8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.281111 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b8483d-1de7-4272-ab23-127c01e065f8 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.282313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 278/555] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:19 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:19.283618 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408a15eb-cdce-4ecb-b74d-5a4ded62ceb5 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.283727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408a15eb-cdce-4ecb-b74d-5a4ded62ceb5 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.283960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408a15eb-cdce-4ecb-b74d-5a4ded62ceb5 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.285390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 279/556] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:19 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:19.286216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408a15eb-cdce-4ecb-b74d-5a4ded62ceb5 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:19.319621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:19.330751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c13c857-8715-4dce-9a1a-0d9f6cc1e71a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:19.332860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c13c857-8715-4dce-9a1a-0d9f6cc1e71a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:19.402687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408a15eb-cdce-4ecb-b74d-5a4ded62ceb5 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:19.429306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:19.437876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd3e9e6-ca62-4ec7-87f6-4b03ae1b44ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:19.438478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd3e9e6-ca62-4ec7-87f6-4b03ae1b44ff None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.438753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd3e9e6-ca62-4ec7-87f6-4b03ae1b44ff None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.439021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd3e9e6-ca62-4ec7-87f6-4b03ae1b44ff None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.439660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd3e9e6-ca62-4ec7-87f6-4b03ae1b44ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:19.439961 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd3e9e6-ca62-4ec7-87f6-4b03ae1b44ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:19.441369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd3e9e6-ca62-4ec7-87f6-4b03ae1b44ff None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:19.447736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 278/557] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:19.449265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1520b544-bba7-4c49-8032-433ced308564 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.449548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1520b544-bba7-4c49-8032-433ced308564 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.449821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1520b544-bba7-4c49-8032-433ced308564 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.457535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 280/558] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:19.458766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d523816-0a12-4927-827d-ca346d8cd593 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.459052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d523816-0a12-4927-827d-ca346d8cd593 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.459536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d523816-0a12-4927-827d-ca346d8cd593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.469404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1520b544-bba7-4c49-8032-433ced308564 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:19.470582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1520b544-bba7-4c49-8032-433ced308564 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:19.478332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-1d523816-0a12-4927-827d-ca346d8cd593 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:19.479496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1d523816-0a12-4927-827d-ca346d8cd593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:19.536611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 279/559] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:19.537774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134872c4-617c-48e0-9848-0de0d4c87ba1 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.538148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134872c4-617c-48e0-9848-0de0d4c87ba1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.538349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134872c4-617c-48e0-9848-0de0d4c87ba1 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.539701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 280/560] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:19 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:19.540284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134872c4-617c-48e0-9848-0de0d4c87ba1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:19.551070 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 281/561] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:19.551760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d523816-0a12-4927-827d-ca346d8cd593 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:19.576078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:19.671518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d523816-0a12-4927-827d-ca346d8cd593 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:19.680857 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134872c4-617c-48e0-9848-0de0d4c87ba1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:19.706734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:19.710276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed41c67-cdcc-4292-8e02-ba74606297be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:19.710912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed41c67-cdcc-4292-8e02-ba74606297be None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.711194 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed41c67-cdcc-4292-8e02-ba74606297be None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.711874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed41c67-cdcc-4292-8e02-ba74606297be None admin] PATH_INFO: `/v3/users/nova` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.721160 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed41c67-cdcc-4292-8e02-ba74606297be None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:19.722442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed41c67-cdcc-4292-8e02-ba74606297be None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:19.734118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2fc831-4a48-45d4-bc37-e7e070f214e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:19.735060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2fc831-4a48-45d4-bc37-e7e070f214e7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.735598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2fc831-4a48-45d4-bc37-e7e070f214e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-bed41c67-cdcc-4292-8e02-ba74606297be None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:19.735873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:19.736842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:19.737859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:19.737859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:19.737859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:19.737859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:19.737859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:19.737859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:19.737859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:19.737859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:19.737859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:19.737859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:19.739325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2fc831-4a48-45d4-bc37-e7e070f214e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.740691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 282/562] 10.222.0.25 () {56 vars in 1093 bytes} [Mon Sep 26 01:23:19 2022] GET /identity/v3/users/nova => generated 82 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:19.741873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d847e6e-208d-498a-9fb2-c74a32d52225 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.742155 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d847e6e-208d-498a-9fb2-c74a32d52225 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.742441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d847e6e-208d-498a-9fb2-c74a32d52225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.756214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2fc831-4a48-45d4-bc37-e7e070f214e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:19.758677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2fc831-4a48-45d4-bc37-e7e070f214e7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:19.763382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-1d847e6e-208d-498a-9fb2-c74a32d52225 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:19.764639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1d847e6e-208d-498a-9fb2-c74a32d52225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-af2fc831-4a48-45d4-bc37-e7e070f214e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:19.778109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:19.779113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:19.780016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:19.780016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:19.780016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:19.782039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 281/563] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:19.783401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2fc831-4a48-45d4-bc37-e7e070f214e7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:19.849775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 283/564] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:19.850613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d847e6e-208d-498a-9fb2-c74a32d52225 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:19.861436 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:19.897838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2fc831-4a48-45d4-bc37-e7e070f214e7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:19.932391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4509aaa1-a479-4bff-86bd-de15f4ee37f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:19.932990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4509aaa1-a479-4bff-86bd-de15f4ee37f6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.933268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4509aaa1-a479-4bff-86bd-de15f4ee37f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.933541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4509aaa1-a479-4bff-86bd-de15f4ee37f6 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.934264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4509aaa1-a479-4bff-86bd-de15f4ee37f6 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:19.934570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4509aaa1-a479-4bff-86bd-de15f4ee37f6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:19.935995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4509aaa1-a479-4bff-86bd-de15f4ee37f6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:19.953403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 282/565] 10.222.0.25 () {56 vars in 1133 bytes} [Mon Sep 26 01:23:19 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:19.954258 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4509aaa1-a479-4bff-86bd-de15f4ee37f6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:19.957838 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d847e6e-208d-498a-9fb2-c74a32d52225 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:19.993027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:19.997796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36455d04-5cc3-4233-90aa-1614a9e8966d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:19.998455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36455d04-5cc3-4233-90aa-1614a9e8966d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:19.998738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36455d04-5cc3-4233-90aa-1614a9e8966d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:19.999007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36455d04-5cc3-4233-90aa-1614a9e8966d None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:19.999643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36455d04-5cc3-4233-90aa-1614a9e8966d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:19.999944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36455d04-5cc3-4233-90aa-1614a9e8966d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:20.001333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36455d04-5cc3-4233-90aa-1614a9e8966d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:20.018010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 284/566] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:20.018010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36455d04-5cc3-4233-90aa-1614a9e8966d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.084974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4509aaa1-a479-4bff-86bd-de15f4ee37f6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.096755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:20.122406 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fcf69a-af6f-49b3-8041-d6bca3f462cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.123017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fcf69a-af6f-49b3-8041-d6bca3f462cd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.123224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fcf69a-af6f-49b3-8041-d6bca3f462cd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.123492 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40fcf69a-af6f-49b3-8041-d6bca3f462cd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.133407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36455d04-5cc3-4233-90aa-1614a9e8966d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.134326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40fcf69a-af6f-49b3-8041-d6bca3f462cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-40fcf69a-af6f-49b3-8041-d6bca3f462cd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:20.139363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:20.140566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:20.141511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:20.145317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 283/567] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:20.145905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fcf69a-af6f-49b3-8041-d6bca3f462cd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.166857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627fdaff-040d-4a49-ba56-e2afd75e12a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.167483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627fdaff-040d-4a49-ba56-e2afd75e12a7 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.167761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627fdaff-040d-4a49-ba56-e2afd75e12a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.168034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627fdaff-040d-4a49-ba56-e2afd75e12a7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.177955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627fdaff-040d-4a49-ba56-e2afd75e12a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-627fdaff-040d-4a49-ba56-e2afd75e12a7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:20.183999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:20.185042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:20.185882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 285/568] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:20.187215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627fdaff-040d-4a49-ba56-e2afd75e12a7 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.230140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:20.245519 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40fcf69a-af6f-49b3-8041-d6bca3f462cd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.279042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccc34b0-a6dd-4d3b-bf08-dde0ca70d825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.279661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccc34b0-a6dd-4d3b-bf08-dde0ca70d825 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.279940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccc34b0-a6dd-4d3b-bf08-dde0ca70d825 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.280213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccc34b0-a6dd-4d3b-bf08-dde0ca70d825 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.288405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627fdaff-040d-4a49-ba56-e2afd75e12a7 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.290997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccc34b0-a6dd-4d3b-bf08-dde0ca70d825 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:20.293354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccc34b0-a6dd-4d3b-bf08-dde0ca70d825 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-2ccc34b0-a6dd-4d3b-bf08-dde0ca70d825 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:20.307932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:20.309111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:20.310315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:20.310315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:20.310315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:20.310315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 284/569] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:20.310315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccc34b0-a6dd-4d3b-bf08-dde0ca70d825 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.319759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:20.329268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990c8694-8160-4827-931d-4bf0978bca7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.329894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990c8694-8160-4827-931d-4bf0978bca7d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.330174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990c8694-8160-4827-931d-4bf0978bca7d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.330447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990c8694-8160-4827-931d-4bf0978bca7d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.337487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990c8694-8160-4827-931d-4bf0978bca7d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:20.338914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990c8694-8160-4827-931d-4bf0978bca7d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-990c8694-8160-4827-931d-4bf0978bca7d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:20.349852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:20.350891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:20.351805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:20.351805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:20.351805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:20.351805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:20.351805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:20.351805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:20.351805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:20.351805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:20.351805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 286/570] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:20.351805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990c8694-8160-4827-931d-4bf0978bca7d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.419027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccc34b0-a6dd-4d3b-bf08-dde0ca70d825 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.452632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9e1877-ffe9-428e-a910-bdda1d92c94f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.453017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990c8694-8160-4827-931d-4bf0978bca7d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.453257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9e1877-ffe9-428e-a910-bdda1d92c94f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.453535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9e1877-ffe9-428e-a910-bdda1d92c94f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.453804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9e1877-ffe9-428e-a910-bdda1d92c94f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.460703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:20.465525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9e1877-ffe9-428e-a910-bdda1d92c94f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-5b9e1877-ffe9-428e-a910-bdda1d92c94f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:20.470478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:20.476763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 285/571] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:20.478465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9e1877-ffe9-428e-a910-bdda1d92c94f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.492544 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1594710a-a821-4b06-ad3b-4feb71d7232f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.493786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1594710a-a821-4b06-ad3b-4feb71d7232f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.494057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1594710a-a821-4b06-ad3b-4feb71d7232f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.494332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1594710a-a821-4b06-ad3b-4feb71d7232f None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.494985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1594710a-a821-4b06-ad3b-4feb71d7232f None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:20.495395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1594710a-a821-4b06-ad3b-4feb71d7232f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:20.496781 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1594710a-a821-4b06-ad3b-4feb71d7232f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:20.504520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 287/572] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:20.505290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1594710a-a821-4b06-ad3b-4feb71d7232f None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.556110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:20.572302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9e1877-ffe9-428e-a910-bdda1d92c94f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.607048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1594710a-a821-4b06-ad3b-4feb71d7232f None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.607215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd7444-d649-45ff-90a4-54ebfa19fe45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.607915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd7444-d649-45ff-90a4-54ebfa19fe45 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.608263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd7444-d649-45ff-90a4-54ebfa19fe45 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.608630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd7444-d649-45ff-90a4-54ebfa19fe45 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.609347 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bd7444-d649-45ff-90a4-54ebfa19fe45 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:20.609711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bd7444-d649-45ff-90a4-54ebfa19fe45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:20.611138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bd7444-d649-45ff-90a4-54ebfa19fe45 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:20.619740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 286/573] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:20.620456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd7444-d649-45ff-90a4-54ebfa19fe45 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.645937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2994efa3-0c75-48a2-b313-7fae83967abe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.646536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2994efa3-0c75-48a2-b313-7fae83967abe None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.646815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2994efa3-0c75-48a2-b313-7fae83967abe None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.647093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2994efa3-0c75-48a2-b313-7fae83967abe None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.647725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2994efa3-0c75-48a2-b313-7fae83967abe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:20.648026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2994efa3-0c75-48a2-b313-7fae83967abe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:20.650056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2994efa3-0c75-48a2-b313-7fae83967abe None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:20.661583 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 288/574] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:20.662366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2994efa3-0c75-48a2-b313-7fae83967abe None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.680664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:20.721974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd7444-d649-45ff-90a4-54ebfa19fe45 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.754405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b1e7eb-e593-4870-b70f-44ad3b1a3575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.754993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b1e7eb-e593-4870-b70f-44ad3b1a3575 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.755269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b1e7eb-e593-4870-b70f-44ad3b1a3575 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.755546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b1e7eb-e593-4870-b70f-44ad3b1a3575 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.756224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b1e7eb-e593-4870-b70f-44ad3b1a3575 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:20.756556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b1e7eb-e593-4870-b70f-44ad3b1a3575 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:20.757976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b1e7eb-e593-4870-b70f-44ad3b1a3575 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:20.766907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2994efa3-0c75-48a2-b313-7fae83967abe None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.769523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 287/575] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:20.770311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b1e7eb-e593-4870-b70f-44ad3b1a3575 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.778595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:20.803118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1322e792-0e13-437c-adc0-76a6c62d7b34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.803715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1322e792-0e13-437c-adc0-76a6c62d7b34 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.803991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1322e792-0e13-437c-adc0-76a6c62d7b34 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.804258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1322e792-0e13-437c-adc0-76a6c62d7b34 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.804943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1322e792-0e13-437c-adc0-76a6c62d7b34 None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:20.805240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1322e792-0e13-437c-adc0-76a6c62d7b34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:20.807191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1322e792-0e13-437c-adc0-76a6c62d7b34 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:20.814850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 289/576] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:20.815621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1322e792-0e13-437c-adc0-76a6c62d7b34 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.869502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b1e7eb-e593-4870-b70f-44ad3b1a3575 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.888894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:20.903442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4f2f94-a100-4505-b5de-d0f60f7fe289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.904147 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4f2f94-a100-4505-b5de-d0f60f7fe289 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.904528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4f2f94-a100-4505-b5de-d0f60f7fe289 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.904806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4f2f94-a100-4505-b5de-d0f60f7fe289 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.914708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4f2f94-a100-4505-b5de-d0f60f7fe289 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:20.916489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4f2f94-a100-4505-b5de-d0f60f7fe289 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:20.918802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1322e792-0e13-437c-adc0-76a6c62d7b34 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-fd4f2f94-a100-4505-b5de-d0f60f7fe289 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:20.931775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:20.932890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:20.933803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:20.933803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:20.933803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:20.933803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 288/577] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:20.934024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4f2f94-a100-4505-b5de-d0f60f7fe289 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:20.958153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8332502-c095-4ccc-b904-d2b2ef7b3483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:20.958844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8332502-c095-4ccc-b904-d2b2ef7b3483 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:20.959116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8332502-c095-4ccc-b904-d2b2ef7b3483 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:20.959383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8332502-c095-4ccc-b904-d2b2ef7b3483 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:20.970250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8332502-c095-4ccc-b904-d2b2ef7b3483 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:20.971700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8332502-c095-4ccc-b904-d2b2ef7b3483 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-f8332502-c095-4ccc-b904-d2b2ef7b3483 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:20.985894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:20.986824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:20.987707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:20.987707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:20.987707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:20.987707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 290/578] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:20.988218 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8332502-c095-4ccc-b904-d2b2ef7b3483 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.013376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:21.036928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4f2f94-a100-4505-b5de-d0f60f7fe289 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.072430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d256439-36c5-4d58-992e-add7dac9e2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.073058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d256439-36c5-4d58-992e-add7dac9e2d2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.073334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d256439-36c5-4d58-992e-add7dac9e2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.073604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d256439-36c5-4d58-992e-add7dac9e2d2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:21.080061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d256439-36c5-4d58-992e-add7dac9e2d2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:21.081812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d256439-36c5-4d58-992e-add7dac9e2d2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:21.089756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8332502-c095-4ccc-b904-d2b2ef7b3483 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-9d256439-36c5-4d58-992e-add7dac9e2d2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:21.093392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:21.094416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:21.095340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:21.095340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:21.095340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:21.095340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:21.095340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:21.095340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:21.095340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:21.095340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:21.095340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 289/579] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:21.095835 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d256439-36c5-4d58-992e-add7dac9e2d2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.125837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:21.126152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b143a67f-65a9-4752-86e2-f0269ccc2013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.126755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b143a67f-65a9-4752-86e2-f0269ccc2013 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.127040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b143a67f-65a9-4752-86e2-f0269ccc2013 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.127312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b143a67f-65a9-4752-86e2-f0269ccc2013 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:21.128117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b143a67f-65a9-4752-86e2-f0269ccc2013 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=e542b14838e2466fb59d17e9a20682dd) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:21.128480 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b143a67f-65a9-4752-86e2-f0269ccc2013 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:21.129965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b143a67f-65a9-4752-86e2-f0269ccc2013 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:21.136004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 291/580] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:20 2022] GET /identity/v3/role_assignments?user.id=e542b14838e2466fb59d17e9a20682dd&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 259 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:21.136637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b143a67f-65a9-4752-86e2-f0269ccc2013 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.220113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d256439-36c5-4d58-992e-add7dac9e2d2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.237095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:21.252721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b143a67f-65a9-4752-86e2-f0269ccc2013 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.261535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70bb5d2-c8db-447b-b741-3351159ecd6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.262268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70bb5d2-c8db-447b-b741-3351159ecd6a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.262642 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70bb5d2-c8db-447b-b741-3351159ecd6a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.262918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70bb5d2-c8db-447b-b741-3351159ecd6a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:21.274801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70bb5d2-c8db-447b-b741-3351159ecd6a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:21.276305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70bb5d2-c8db-447b-b741-3351159ecd6a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:21.292205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531aac48-a240-4156-b0ee-2f389ae313eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.292914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531aac48-a240-4156-b0ee-2f389ae313eb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d70bb5d2-c8db-447b-b741-3351159ecd6a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:21.293042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:21.293928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:21.299877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:21.299877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:21.299877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:21.300051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531aac48-a240-4156-b0ee-2f389ae313eb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.300051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531aac48-a240-4156-b0ee-2f389ae313eb None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:21.300051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531aac48-a240-4156-b0ee-2f389ae313eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:21.300051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531aac48-a240-4156-b0ee-2f389ae313eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:21.300300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 290/581] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:21.300300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70bb5d2-c8db-447b-b741-3351159ecd6a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.300460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531aac48-a240-4156-b0ee-2f389ae313eb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:21.313542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 292/582] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:21.314426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531aac48-a240-4156-b0ee-2f389ae313eb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.394635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:21.411827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70bb5d2-c8db-447b-b741-3351159ecd6a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.423991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531aac48-a240-4156-b0ee-2f389ae313eb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.445932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2fb379-da1c-4bd7-9efb-3556c3711f97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.447093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2fb379-da1c-4bd7-9efb-3556c3711f97 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.447396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2fb379-da1c-4bd7-9efb-3556c3711f97 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.447741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2fb379-da1c-4bd7-9efb-3556c3711f97 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:21.448487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2fb379-da1c-4bd7-9efb-3556c3711f97 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:21.448797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2fb379-da1c-4bd7-9efb-3556c3711f97 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:21.450169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2fb379-da1c-4bd7-9efb-3556c3711f97 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:21.461145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7604660-502b-4d8e-90b8-a713cf5fe2c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.462047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7604660-502b-4d8e-90b8-a713cf5fe2c1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.466804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 291/583] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:21.466804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2fb379-da1c-4bd7-9efb-3556c3711f97 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.467023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7604660-502b-4d8e-90b8-a713cf5fe2c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.467023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7604660-502b-4d8e-90b8-a713cf5fe2c1 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:21.467023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7604660-502b-4d8e-90b8-a713cf5fe2c1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:21.467023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7604660-502b-4d8e-90b8-a713cf5fe2c1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:21.467460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7604660-502b-4d8e-90b8-a713cf5fe2c1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:21.485266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 293/584] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:21.485266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7604660-502b-4d8e-90b8-a713cf5fe2c1 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.492431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:21.591008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2fb379-da1c-4bd7-9efb-3556c3711f97 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.601718 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7604660-502b-4d8e-90b8-a713cf5fe2c1 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.625995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b012a16c-d011-4bd5-a7f3-d5bd44dd3c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.626718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b012a16c-d011-4bd5-a7f3-d5bd44dd3c9e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.626999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b012a16c-d011-4bd5-a7f3-d5bd44dd3c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.627276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b012a16c-d011-4bd5-a7f3-d5bd44dd3c9e None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:21.627917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b012a16c-d011-4bd5-a7f3-d5bd44dd3c9e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:21.628225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b012a16c-d011-4bd5-a7f3-d5bd44dd3c9e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:21.629776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b012a16c-d011-4bd5-a7f3-d5bd44dd3c9e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:21.640744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea9b075-9204-4080-ab45-98569e2dd3b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.641369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea9b075-9204-4080-ab45-98569e2dd3b0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.641646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea9b075-9204-4080-ab45-98569e2dd3b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.641916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea9b075-9204-4080-ab45-98569e2dd3b0 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:21.647774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 292/585] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:21.647774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b012a16c-d011-4bd5-a7f3-d5bd44dd3c9e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.651915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:21.654344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea9b075-9204-4080-ab45-98569e2dd3b0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:21.655881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea9b075-9204-4080-ab45-98569e2dd3b0 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-8ea9b075-9204-4080-ab45-98569e2dd3b0 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:21.667929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:21.669224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:21.670271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 294/586] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:21 2022] GET /identity/v3/users/glance => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:21.671016 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea9b075-9204-4080-ab45-98569e2dd3b0 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.745130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:21.760556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b012a16c-d011-4bd5-a7f3-d5bd44dd3c9e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.782313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea9b075-9204-4080-ab45-98569e2dd3b0 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.797187 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71019dd-566a-41bb-96d9-f382a6529a10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.797821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71019dd-566a-41bb-96d9-f382a6529a10 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.798099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71019dd-566a-41bb-96d9-f382a6529a10 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.798374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71019dd-566a-41bb-96d9-f382a6529a10 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:21.807972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71019dd-566a-41bb-96d9-f382a6529a10 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:21.809459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71019dd-566a-41bb-96d9-f382a6529a10 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:21.819017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8966db7c-0d9c-496b-a058-bf19637ded3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.819682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8966db7c-0d9c-496b-a058-bf19637ded3e None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.819963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8966db7c-0d9c-496b-a058-bf19637ded3e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.820240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8966db7c-0d9c-496b-a058-bf19637ded3e None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/381ad7c239f444da90369dc58779b86c/roles/03dc4e232ede47a699064dda38338428` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-c71019dd-566a-41bb-96d9-f382a6529a10 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:21.820906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:21.822107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 293/587] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:21 2022] GET /identity/v3/users/cinder => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:21.823049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71019dd-566a-41bb-96d9-f382a6529a10 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.842439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8966db7c-0d9c-496b-a058-bf19637ded3e None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=381ad7c239f444da90369dc58779b86c, role_id=03dc4e232ede47a699064dda38338428)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:21.843888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8966db7c-0d9c-496b-a058-bf19637ded3e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:21.867564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 295/588] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:23:21 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/381ad7c239f444da90369dc58779b86c/roles/03dc4e232ede47a699064dda38338428 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:23:21.869463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8966db7c-0d9c-496b-a058-bf19637ded3e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:21.895422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:21.942328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71019dd-566a-41bb-96d9-f382a6529a10 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:21.989280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a95eda-4e77-4269-ba30-088fb222f9d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:21.991004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a95eda-4e77-4269-ba30-088fb222f9d5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:21.991531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a95eda-4e77-4269-ba30-088fb222f9d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:21.992014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a95eda-4e77-4269-ba30-088fb222f9d5 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.005005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a95eda-4e77-4269-ba30-088fb222f9d5 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.007868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a95eda-4e77-4269-ba30-088fb222f9d5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:22.020425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8966db7c-0d9c-496b-a058-bf19637ded3e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-06a95eda-4e77-4269-ba30-088fb222f9d5 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:22.027020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:22.028067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:22.029168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:22.029168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:22.029168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:22.029168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:22.029168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:22.029168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:22.029168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:22.029168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:22.029168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 26 01:23:22.029168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:22.036837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 294/589] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:21 2022] GET /identity/v3/users/neutron => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:22.038003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a95eda-4e77-4269-ba30-088fb222f9d5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:22.044148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:22.064574 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0d3a7c-35ed-47df-a0f4-c6337adce364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:22.064795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0d3a7c-35ed-47df-a0f4-c6337adce364 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:22.065075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0d3a7c-35ed-47df-a0f4-c6337adce364 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:22.065355 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0d3a7c-35ed-47df-a0f4-c6337adce364 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.066085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0d3a7c-35ed-47df-a0f4-c6337adce364 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:22.066386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0d3a7c-35ed-47df-a0f4-c6337adce364 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.067881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0d3a7c-35ed-47df-a0f4-c6337adce364 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:22.086476 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 296/590] 10.222.0.25 () {56 vars in 1137 bytes} [Mon Sep 26 01:23:21 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:22.088394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0d3a7c-35ed-47df-a0f4-c6337adce364 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:22.166455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a95eda-4e77-4269-ba30-088fb222f9d5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:22.192098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:22.204269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba1e861-1c80-4e8c-9c5f-0fb1b7ccf979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:22.205042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba1e861-1c80-4e8c-9c5f-0fb1b7ccf979 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:22.205459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba1e861-1c80-4e8c-9c5f-0fb1b7ccf979 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:22.205755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba1e861-1c80-4e8c-9c5f-0fb1b7ccf979 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.206756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba1e861-1c80-4e8c-9c5f-0fb1b7ccf979 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:22.207068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba1e861-1c80-4e8c-9c5f-0fb1b7ccf979 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.209032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba1e861-1c80-4e8c-9c5f-0fb1b7ccf979 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:22.219562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0d3a7c-35ed-47df-a0f4-c6337adce364 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:22.231415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 295/591] 10.222.0.25 () {56 vars in 1137 bytes} [Mon Sep 26 01:23:22 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:22.232281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba1e861-1c80-4e8c-9c5f-0fb1b7ccf979 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:22.269757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a965ec-3e86-4ae4-bc43-1a5feeabac35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:22.270968 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a965ec-3e86-4ae4-bc43-1a5feeabac35 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:22.271662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a965ec-3e86-4ae4-bc43-1a5feeabac35 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:22.272466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a965ec-3e86-4ae4-bc43-1a5feeabac35 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.273710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a965ec-3e86-4ae4-bc43-1a5feeabac35 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:22.274826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a965ec-3e86-4ae4-bc43-1a5feeabac35 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.277403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a965ec-3e86-4ae4-bc43-1a5feeabac35 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:22.303729 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 297/592] 10.222.0.25 () {56 vars in 1139 bytes} [Mon Sep 26 01:23:22 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:22.304654 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a965ec-3e86-4ae4-bc43-1a5feeabac35 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:22.342301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:22.357685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba1e861-1c80-4e8c-9c5f-0fb1b7ccf979 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:22.393857 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c36c79-dfe2-46a6-8b8e-757c50fecad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:22.394875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c36c79-dfe2-46a6-8b8e-757c50fecad0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:22.395388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c36c79-dfe2-46a6-8b8e-757c50fecad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:22.395938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c36c79-dfe2-46a6-8b8e-757c50fecad0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.407628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c36c79-dfe2-46a6-8b8e-757c50fecad0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.409263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c36c79-dfe2-46a6-8b8e-757c50fecad0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:22.417878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a965ec-3e86-4ae4-bc43-1a5feeabac35 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-56c36c79-dfe2-46a6-8b8e-757c50fecad0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:22.428451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:22.429518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:22.431116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:22.431116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:22.431116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:22.436430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 296/593] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:22.437490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c36c79-dfe2-46a6-8b8e-757c50fecad0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:22.470922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1292695a-dc1e-4433-b488-c54e428249b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:22.471815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292695a-dc1e-4433-b488-c54e428249b1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:22.472450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292695a-dc1e-4433-b488-c54e428249b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:22.472800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292695a-dc1e-4433-b488-c54e428249b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.476480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:22.488067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1292695a-dc1e-4433-b488-c54e428249b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.490382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1292695a-dc1e-4433-b488-c54e428249b1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-1292695a-dc1e-4433-b488-c54e428249b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:22.512490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:22.513665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:22.514798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:22.514798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:22.514798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:22.517031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 298/594] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:22.518157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1292695a-dc1e-4433-b488-c54e428249b1 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:22.555818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c36c79-dfe2-46a6-8b8e-757c50fecad0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:22.591982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f03f52-d248-4fd1-bfbf-b53b916068dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:22.592799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f03f52-d248-4fd1-bfbf-b53b916068dd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:22.593181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f03f52-d248-4fd1-bfbf-b53b916068dd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:22.593558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f03f52-d248-4fd1-bfbf-b53b916068dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.607102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f03f52-d248-4fd1-bfbf-b53b916068dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.608731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f03f52-d248-4fd1-bfbf-b53b916068dd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a0f03f52-d248-4fd1-bfbf-b53b916068dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:22.624820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:22.625920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:22.625920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:22.625920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:22.626930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 297/595] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:22.632306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f03f52-d248-4fd1-bfbf-b53b916068dd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:22.638760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:22.659677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1292695a-dc1e-4433-b488-c54e428249b1 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:22.706301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23858d70-d5d6-47d0-aedf-a410c59b2ad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:22.707443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23858d70-d5d6-47d0-aedf-a410c59b2ad7 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:22.707889 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23858d70-d5d6-47d0-aedf-a410c59b2ad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:22.708496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23858d70-d5d6-47d0-aedf-a410c59b2ad7 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.709792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23858d70-d5d6-47d0-aedf-a410c59b2ad7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:22.710265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23858d70-d5d6-47d0-aedf-a410c59b2ad7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.712171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23858d70-d5d6-47d0-aedf-a410c59b2ad7 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:22.733324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 299/596] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:22.733324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23858d70-d5d6-47d0-aedf-a410c59b2ad7 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:22.734926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:22.753247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f03f52-d248-4fd1-bfbf-b53b916068dd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:22.791405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ecbafd-dff2-4222-af03-c63b65f0a5e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:22.792228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ecbafd-dff2-4222-af03-c63b65f0a5e0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:22.792644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ecbafd-dff2-4222-af03-c63b65f0a5e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:22.793015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ecbafd-dff2-4222-af03-c63b65f0a5e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.794050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ecbafd-dff2-4222-af03-c63b65f0a5e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:22.794260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ecbafd-dff2-4222-af03-c63b65f0a5e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.795733 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ecbafd-dff2-4222-af03-c63b65f0a5e0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:22.814302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 298/597] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:22.815828 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ecbafd-dff2-4222-af03-c63b65f0a5e0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:22.879208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23858d70-d5d6-47d0-aedf-a410c59b2ad7 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:22.925647 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d323caf3-86ac-4fa4-9204-3be2be9d87ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:22.926532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d323caf3-86ac-4fa4-9204-3be2be9d87ee None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:22.926957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d323caf3-86ac-4fa4-9204-3be2be9d87ee None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:22.927271 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ecbafd-dff2-4222-af03-c63b65f0a5e0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:22.927555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d323caf3-86ac-4fa4-9204-3be2be9d87ee None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.928334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d323caf3-86ac-4fa4-9204-3be2be9d87ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:22.928821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d323caf3-86ac-4fa4-9204-3be2be9d87ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.932165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d323caf3-86ac-4fa4-9204-3be2be9d87ee None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:22.947190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 300/598] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:22.948427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d323caf3-86ac-4fa4-9204-3be2be9d87ee None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:22.962419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:22.973746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6f6a48-52a2-4477-859c-ebccd094abe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:22.974997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6f6a48-52a2-4477-859c-ebccd094abe3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:22.975940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6f6a48-52a2-4477-859c-ebccd094abe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:22.976436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6f6a48-52a2-4477-859c-ebccd094abe3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:22.986033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6f6a48-52a2-4477-859c-ebccd094abe3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:22.988631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6f6a48-52a2-4477-859c-ebccd094abe3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-da6f6a48-52a2-4477-859c-ebccd094abe3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:23.004558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:23.005503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:23.006430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:23.006430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:23.006430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:23.006430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:23.006430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:23.006430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:23.006430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:23.006430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:23.009283 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 299/599] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:23.010240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6f6a48-52a2-4477-859c-ebccd094abe3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:23.024891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:23.088330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d323caf3-86ac-4fa4-9204-3be2be9d87ee None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:23.124533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356f00e9-75ec-4532-a878-5102bfab411a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:23.125419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356f00e9-75ec-4532-a878-5102bfab411a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:23.125899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356f00e9-75ec-4532-a878-5102bfab411a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:23.126365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356f00e9-75ec-4532-a878-5102bfab411a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:23.133712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356f00e9-75ec-4532-a878-5102bfab411a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:23.135494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356f00e9-75ec-4532-a878-5102bfab411a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-356f00e9-75ec-4532-a878-5102bfab411a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:23.147963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:23.148936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:23.150441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:23.150441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:23.150441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:23.150441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:23.150441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:23.150441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:23.150441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:23.150441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:23.157185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6f6a48-52a2-4477-859c-ebccd094abe3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:23.157471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 301/600] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:23.158516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356f00e9-75ec-4532-a878-5102bfab411a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:23.199870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c11300-4a85-4cbb-9d4d-202a24d0f65c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:23.200531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c11300-4a85-4cbb-9d4d-202a24d0f65c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:23.200813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c11300-4a85-4cbb-9d4d-202a24d0f65c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:23.201084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c11300-4a85-4cbb-9d4d-202a24d0f65c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:23.212904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c11300-4a85-4cbb-9d4d-202a24d0f65c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:23.214713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c11300-4a85-4cbb-9d4d-202a24d0f65c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-65c11300-4a85-4cbb-9d4d-202a24d0f65c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:23.232409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:23.233426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:23.234331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:23.234331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:23.234331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:23.234331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:23.234331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:23.234331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:23.234331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:23.234331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:23.236442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 300/601] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:23.237417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c11300-4a85-4cbb-9d4d-202a24d0f65c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:23.238987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:23.295333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356f00e9-75ec-4532-a878-5102bfab411a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:23.310822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:23.346542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735b5738-3bc7-41a8-88a9-7df9cdac82c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:23.347066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735b5738-3bc7-41a8-88a9-7df9cdac82c6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:23.347335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735b5738-3bc7-41a8-88a9-7df9cdac82c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:23.347605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735b5738-3bc7-41a8-88a9-7df9cdac82c6 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:23.347762 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c11300-4a85-4cbb-9d4d-202a24d0f65c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:23.348293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735b5738-3bc7-41a8-88a9-7df9cdac82c6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:23.348628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735b5738-3bc7-41a8-88a9-7df9cdac82c6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:23.350055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735b5738-3bc7-41a8-88a9-7df9cdac82c6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:23.362846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 302/602] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:23.363653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735b5738-3bc7-41a8-88a9-7df9cdac82c6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:23.392101 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fb7cf4-2b06-403c-895b-866b0aa198ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:23.393461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fb7cf4-2b06-403c-895b-866b0aa198ac None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:23.394066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fb7cf4-2b06-403c-895b-866b0aa198ac None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:23.394665 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fb7cf4-2b06-403c-895b-866b0aa198ac None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:23.395484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fb7cf4-2b06-403c-895b-866b0aa198ac None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:23.395864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fb7cf4-2b06-403c-895b-866b0aa198ac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:23.397399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fb7cf4-2b06-403c-895b-866b0aa198ac None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:23.411940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 301/603] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:23.413174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d25654-2f91-4ae9-86d4-000f896bd927 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:23.413458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d25654-2f91-4ae9-86d4-000f896bd927 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:23.413730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d25654-2f91-4ae9-86d4-000f896bd927 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:23.419151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 302/604] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:23.419151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d25654-2f91-4ae9-86d4-000f896bd927 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:23.478693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735b5738-3bc7-41a8-88a9-7df9cdac82c6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:23.507502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:23.518778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c633ca3-e66e-44b0-a2fb-02222ed4d253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:23.519553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c633ca3-e66e-44b0-a2fb-02222ed4d253 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:23.519683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c633ca3-e66e-44b0-a2fb-02222ed4d253 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:23.519914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c633ca3-e66e-44b0-a2fb-02222ed4d253 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:23.520636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c633ca3-e66e-44b0-a2fb-02222ed4d253 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:23.520963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c633ca3-e66e-44b0-a2fb-02222ed4d253 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:23.522432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c633ca3-e66e-44b0-a2fb-02222ed4d253 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:23.523542 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d25654-2f91-4ae9-86d4-000f896bd927 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:23.540309 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 303/605] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:23.541228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c633ca3-e66e-44b0-a2fb-02222ed4d253 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:23.551805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:23.561281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e723be0-e9a0-4439-9e3b-0284892562ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:23.561927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e723be0-e9a0-4439-9e3b-0284892562ae None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:23.562206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e723be0-e9a0-4439-9e3b-0284892562ae None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:23.562479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e723be0-e9a0-4439-9e3b-0284892562ae None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/e07b6890d3004dd78faed48a1ab90086/roles/03dc4e232ede47a699064dda38338428` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:23.577714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e723be0-e9a0-4439-9e3b-0284892562ae None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=e07b6890d3004dd78faed48a1ab90086, role_id=03dc4e232ede47a699064dda38338428)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:23.579533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e723be0-e9a0-4439-9e3b-0284892562ae None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:23.592977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 303/606] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:23:23 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/e07b6890d3004dd78faed48a1ab90086/roles/03dc4e232ede47a699064dda38338428 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:23:23.605881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf4804a-ca14-4926-8137-0344c9b6b484 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:23.606096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf4804a-ca14-4926-8137-0344c9b6b484 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:23.606375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf4804a-ca14-4926-8137-0344c9b6b484 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:23.625873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-edf4804a-ca14-4926-8137-0344c9b6b484 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:23.627120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-edf4804a-ca14-4926-8137-0344c9b6b484 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:23.681504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c633ca3-e66e-44b0-a2fb-02222ed4d253 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:23.719873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 304/607] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:23.720811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf4804a-ca14-4926-8137-0344c9b6b484 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:23.737103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c96f4a-367e-4e72-9735-2bc3586cf020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:23.737792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c96f4a-367e-4e72-9735-2bc3586cf020 None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:23.738072 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c96f4a-367e-4e72-9735-2bc3586cf020 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:23.738469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c96f4a-367e-4e72-9735-2bc3586cf020 None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/2bb5f76eb08741f3a8c7a3a140e01e24/roles/03dc4e232ede47a699064dda38338428` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:23.773696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c96f4a-367e-4e72-9735-2bc3586cf020 None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=2bb5f76eb08741f3a8c7a3a140e01e24, role_id=03dc4e232ede47a699064dda38338428)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:23.775300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c96f4a-367e-4e72-9735-2bc3586cf020 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:23.796575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 304/608] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:23:23 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/2bb5f76eb08741f3a8c7a3a140e01e24/roles/03dc4e232ede47a699064dda38338428 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:23:23.869031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf4804a-ca14-4926-8137-0344c9b6b484 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:23.877442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:23.911190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5f560e-af2d-478d-906c-61f67b93e73b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:23.911868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f560e-af2d-478d-906c-61f67b93e73b None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:23.912151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f560e-af2d-478d-906c-61f67b93e73b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:23.912459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5f560e-af2d-478d-906c-61f67b93e73b None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/0478ebab37fb4f488eef84f7a241e206/roles/03dc4e232ede47a699064dda38338428` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:23.928984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5f560e-af2d-478d-906c-61f67b93e73b None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=0478ebab37fb4f488eef84f7a241e206, role_id=03dc4e232ede47a699064dda38338428)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:23.931079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5f560e-af2d-478d-906c-61f67b93e73b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:23.953313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 305/609] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:23:23 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/0478ebab37fb4f488eef84f7a241e206/roles/03dc4e232ede47a699064dda38338428 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:23:24.069808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c96f4a-367e-4e72-9735-2bc3586cf020 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:24.111770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:24.188386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba32b24-2fe2-400a-a405-cd1c734d96b4 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:24.188681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba32b24-2fe2-400a-a405-cd1c734d96b4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:24.188879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c96f4a-367e-4e72-9735-2bc3586cf020 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:24.188964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba32b24-2fe2-400a-a405-cd1c734d96b4 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:24.190624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 306/610] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:23:24.196017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17c5152-34ee-4b06-af3f-9a3938d51ffd None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:24.196380 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17c5152-34ee-4b06-af3f-9a3938d51ffd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:24.196621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17c5152-34ee-4b06-af3f-9a3938d51ffd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:24.229423 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e17c5152-34ee-4b06-af3f-9a3938d51ffd None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:24.230657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e17c5152-34ee-4b06-af3f-9a3938d51ffd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:24.233511 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5939dd2a-d9df-42e5-a800-22a08cf332b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:24.234147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5939dd2a-d9df-42e5-a800-22a08cf332b3 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:24.234484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5939dd2a-d9df-42e5-a800-22a08cf332b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:24.234784 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5939dd2a-d9df-42e5-a800-22a08cf332b3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:24.247054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5939dd2a-d9df-42e5-a800-22a08cf332b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-5939dd2a-d9df-42e5-a800-22a08cf332b3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:24.252132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:24.253108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:24.254364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 305/611] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:24.259774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5939dd2a-d9df-42e5-a800-22a08cf332b3 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:24.334335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 307/612] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:24.371546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:24.378427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5939dd2a-d9df-42e5-a800-22a08cf332b3 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:24.415503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2682c19-aa4a-41c0-a7d2-7fd387c15207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:24.416527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2682c19-aa4a-41c0-a7d2-7fd387c15207 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:24.417011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2682c19-aa4a-41c0-a7d2-7fd387c15207 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:24.417482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2682c19-aa4a-41c0-a7d2-7fd387c15207 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:24.418346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2682c19-aa4a-41c0-a7d2-7fd387c15207 None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:24.418890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2682c19-aa4a-41c0-a7d2-7fd387c15207 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:24.420491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2682c19-aa4a-41c0-a7d2-7fd387c15207 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:24.431027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 306/613] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:24.436506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17c5152-34ee-4b06-af3f-9a3938d51ffd None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:24.574155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:24.603990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17c5152-34ee-4b06-af3f-9a3938d51ffd None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:24.656250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f256cbce-7334-42f1-9411-a2310205d318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:24.656909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f256cbce-7334-42f1-9411-a2310205d318 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:24.657188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f256cbce-7334-42f1-9411-a2310205d318 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:24.657458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f256cbce-7334-42f1-9411-a2310205d318 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:24.668139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f256cbce-7334-42f1-9411-a2310205d318 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:24.669635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f256cbce-7334-42f1-9411-a2310205d318 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:24.688626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2682c19-aa4a-41c0-a7d2-7fd387c15207 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f256cbce-7334-42f1-9411-a2310205d318 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:24.693628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:24.694573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:24.696083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:24.696083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:24.696083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:24.696083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 308/614] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:24.700539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f256cbce-7334-42f1-9411-a2310205d318 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:24.801562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2682c19-aa4a-41c0-a7d2-7fd387c15207 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:24.839401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048556e7-63e3-4c35-9706-d1a918e0ae9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:24.840383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048556e7-63e3-4c35-9706-d1a918e0ae9c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:24.840888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048556e7-63e3-4c35-9706-d1a918e0ae9c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:24.841373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048556e7-63e3-4c35-9706-d1a918e0ae9c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:24.852989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048556e7-63e3-4c35-9706-d1a918e0ae9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-048556e7-63e3-4c35-9706-d1a918e0ae9c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:24.858846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:24.859778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:24.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:24.867742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 307/615] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:24.869400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048556e7-63e3-4c35-9706-d1a918e0ae9c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:24.890107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:24.895966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f256cbce-7334-42f1-9411-a2310205d318 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:24.974854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048556e7-63e3-4c35-9706-d1a918e0ae9c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:25.001246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e98c9f-4da7-4d50-9e75-ab90217302d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:25.001883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e98c9f-4da7-4d50-9e75-ab90217302d7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:25.002074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e98c9f-4da7-4d50-9e75-ab90217302d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:25.002356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e98c9f-4da7-4d50-9e75-ab90217302d7 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:25.003030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e98c9f-4da7-4d50-9e75-ab90217302d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:25.003337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e98c9f-4da7-4d50-9e75-ab90217302d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:25.004807 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e98c9f-4da7-4d50-9e75-ab90217302d7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:25.014634 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd1501-73d8-40b7-8635-1f30dc50a69c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:25.015639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd1501-73d8-40b7-8635-1f30dc50a69c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:25.016121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd1501-73d8-40b7-8635-1f30dc50a69c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:25.016652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfd1501-73d8-40b7-8635-1f30dc50a69c None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:25.017570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd1501-73d8-40b7-8635-1f30dc50a69c None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:25.018091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd1501-73d8-40b7-8635-1f30dc50a69c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:25.019654 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfd1501-73d8-40b7-8635-1f30dc50a69c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:25.020682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:25.034434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 308/616] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:25.041183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd1501-73d8-40b7-8635-1f30dc50a69c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:25.042531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 309/617] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 342 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:25.049543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e98c9f-4da7-4d50-9e75-ab90217302d7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:25.167232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:25.189640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfd1501-73d8-40b7-8635-1f30dc50a69c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:25.233384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64576bd0-6ca1-4366-a630-44eaef5bd757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:25.234413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64576bd0-6ca1-4366-a630-44eaef5bd757 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:25.236409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e98c9f-4da7-4d50-9e75-ab90217302d7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:25.236843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64576bd0-6ca1-4366-a630-44eaef5bd757 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:25.237297 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64576bd0-6ca1-4366-a630-44eaef5bd757 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:25.258074 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64576bd0-6ca1-4366-a630-44eaef5bd757 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:25.261163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64576bd0-6ca1-4366-a630-44eaef5bd757 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-64576bd0-6ca1-4366-a630-44eaef5bd757 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:25.293976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:25.294996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:25.295884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:25.295884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:25.295884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:25.295884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 309/618] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:25.302280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64576bd0-6ca1-4366-a630-44eaef5bd757 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:25.316624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf0b327-ef25-488d-bed3-37ecc5183b86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:25.317832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf0b327-ef25-488d-bed3-37ecc5183b86 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:25.318110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf0b327-ef25-488d-bed3-37ecc5183b86 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:25.318378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf0b327-ef25-488d-bed3-37ecc5183b86 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:25.332074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf0b327-ef25-488d-bed3-37ecc5183b86 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:25.333806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf0b327-ef25-488d-bed3-37ecc5183b86 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-7bf0b327-ef25-488d-bed3-37ecc5183b86 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:25.347727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:25.348926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:25.349877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 310/619] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:25 2022] GET /identity/v3/users/swift => generated 83 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:25.357189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf0b327-ef25-488d-bed3-37ecc5183b86 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:25.459297 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:25.506999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64576bd0-6ca1-4366-a630-44eaef5bd757 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:25.536926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf0b327-ef25-488d-bed3-37ecc5183b86 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:25.565038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:25.574184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a9cf93-b3b1-41d8-8ff7-9f302cff8508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:25.574800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a9cf93-b3b1-41d8-8ff7-9f302cff8508 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:25.575077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a9cf93-b3b1-41d8-8ff7-9f302cff8508 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:25.575352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a9cf93-b3b1-41d8-8ff7-9f302cff8508 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:25.575997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a9cf93-b3b1-41d8-8ff7-9f302cff8508 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:25.576302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a9cf93-b3b1-41d8-8ff7-9f302cff8508 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:25.577700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a9cf93-b3b1-41d8-8ff7-9f302cff8508 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:25.608421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 310/620] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:25.609375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a9cf93-b3b1-41d8-8ff7-9f302cff8508 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:25.611881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64346e86-0bbd-47e6-98b1-b3e2c5e23e5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:25.612530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64346e86-0bbd-47e6-98b1-b3e2c5e23e5a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:25.612913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64346e86-0bbd-47e6-98b1-b3e2c5e23e5a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:25.613192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64346e86-0bbd-47e6-98b1-b3e2c5e23e5a None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:25.613928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64346e86-0bbd-47e6-98b1-b3e2c5e23e5a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:25.614234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64346e86-0bbd-47e6-98b1-b3e2c5e23e5a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:25.615662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64346e86-0bbd-47e6-98b1-b3e2c5e23e5a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:25.636509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 311/621] 10.222.0.25 () {56 vars in 1135 bytes} [Mon Sep 26 01:23:25 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:25.641961 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64346e86-0bbd-47e6-98b1-b3e2c5e23e5a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:25.762196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64346e86-0bbd-47e6-98b1-b3e2c5e23e5a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:25.792072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:25.805061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1ad594-c802-4843-8177-82e71d33603b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:25.806573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1ad594-c802-4843-8177-82e71d33603b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:25.806953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1ad594-c802-4843-8177-82e71d33603b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:25.807460 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1ad594-c802-4843-8177-82e71d33603b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:25.818760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1ad594-c802-4843-8177-82e71d33603b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:25.820368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1ad594-c802-4843-8177-82e71d33603b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:25.834288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a9cf93-b3b1-41d8-8ff7-9f302cff8508 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-2b1ad594-c802-4843-8177-82e71d33603b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:25.837856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:25.838810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:25.839696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:25.839696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:25.839696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:25.839696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 312/622] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:25.849493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1ad594-c802-4843-8177-82e71d33603b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:25.903352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc974f1-e1be-4f59-939d-6f1a49bbe760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:25.904331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc974f1-e1be-4f59-939d-6f1a49bbe760 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:25.905529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc974f1-e1be-4f59-939d-6f1a49bbe760 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:25.906589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc974f1-e1be-4f59-939d-6f1a49bbe760 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:25.918252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc974f1-e1be-4f59-939d-6f1a49bbe760 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:25.922652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc974f1-e1be-4f59-939d-6f1a49bbe760 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-7cc974f1-e1be-4f59-939d-6f1a49bbe760 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:25.943705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:25.944921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:25.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:25.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:25.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:25.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:25.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:25.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:25.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:25.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:25.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:25.946157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:25.952060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 311/623] 10.222.0.25 () {56 vars in 1093 bytes} [Mon Sep 26 01:23:25 2022] GET /identity/v3/users/nova => generated 82 bytes in 343 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:25.959186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc974f1-e1be-4f59-939d-6f1a49bbe760 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:26.034191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1ad594-c802-4843-8177-82e71d33603b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:26.072016 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:26.101267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c71dc-bffb-4354-87cc-e5862f2fc6da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:26.101874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c71dc-bffb-4354-87cc-e5862f2fc6da None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.102280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c71dc-bffb-4354-87cc-e5862f2fc6da None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.102516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c71dc-bffb-4354-87cc-e5862f2fc6da None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.103171 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588c71dc-bffb-4354-87cc-e5862f2fc6da None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:26.103475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588c71dc-bffb-4354-87cc-e5862f2fc6da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:26.104882 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588c71dc-bffb-4354-87cc-e5862f2fc6da None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:26.127175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 313/624] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:26.133024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c71dc-bffb-4354-87cc-e5862f2fc6da None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:26.170871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc974f1-e1be-4f59-939d-6f1a49bbe760 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:26.176824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:26.254045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30227f5f-0baf-400b-94f9-5ce65708a309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:26.254670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30227f5f-0baf-400b-94f9-5ce65708a309 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.254956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30227f5f-0baf-400b-94f9-5ce65708a309 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.255230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30227f5f-0baf-400b-94f9-5ce65708a309 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.255970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30227f5f-0baf-400b-94f9-5ce65708a309 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:26.256276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30227f5f-0baf-400b-94f9-5ce65708a309 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:26.257427 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c71dc-bffb-4354-87cc-e5862f2fc6da None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:26.257738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30227f5f-0baf-400b-94f9-5ce65708a309 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:26.298475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 312/625] 10.222.0.25 () {56 vars in 1133 bytes} [Mon Sep 26 01:23:25 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:26.299819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6015b3aa-78d6-4f98-8f9d-f1eebddea4eb None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.300159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6015b3aa-78d6-4f98-8f9d-f1eebddea4eb None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.300474 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6015b3aa-78d6-4f98-8f9d-f1eebddea4eb None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.304490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 313/626] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:26 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:26.310660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6015b3aa-78d6-4f98-8f9d-f1eebddea4eb None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:26.319486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97b4707-f19e-440c-9fae-91518a7f834d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:26.320599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97b4707-f19e-440c-9fae-91518a7f834d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.322556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97b4707-f19e-440c-9fae-91518a7f834d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.323086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97b4707-f19e-440c-9fae-91518a7f834d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.334008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97b4707-f19e-440c-9fae-91518a7f834d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:26.336243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97b4707-f19e-440c-9fae-91518a7f834d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d97b4707-f19e-440c-9fae-91518a7f834d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:26.354866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:26.355841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:26.356789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:26.356789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:26.356789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:26.356789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:26.356789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:26.356789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:26.356789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:26.356789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:26.359296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 314/627] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:26.365797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a09c62-af7a-4492-b2c2-e591703b78fc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.366325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a09c62-af7a-4492-b2c2-e591703b78fc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.366803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a09c62-af7a-4492-b2c2-e591703b78fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.386116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-15a09c62-af7a-4492-b2c2-e591703b78fc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:26.387674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-15a09c62-af7a-4492-b2c2-e591703b78fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:26.433176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6015b3aa-78d6-4f98-8f9d-f1eebddea4eb None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:26.445458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:26.460019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 315/628] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:26.461543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a09c62-af7a-4492-b2c2-e591703b78fc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:26.473141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d129a4f-5d39-47bd-851e-fc353a8730c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:26.473492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d129a4f-5d39-47bd-851e-fc353a8730c0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.473784 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d129a4f-5d39-47bd-851e-fc353a8730c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.474043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d129a4f-5d39-47bd-851e-fc353a8730c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.493985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:26.548620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d129a4f-5d39-47bd-851e-fc353a8730c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:26.550102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d129a4f-5d39-47bd-851e-fc353a8730c0 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:26.615214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a09c62-af7a-4492-b2c2-e591703b78fc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:26.662470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5963146d-ed86-4990-b5e5-29e88c3e3a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:26.663012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 314/629] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:26.663621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5963146d-ed86-4990-b5e5-29e88c3e3a2e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.664099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5963146d-ed86-4990-b5e5-29e88c3e3a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.664590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5963146d-ed86-4990-b5e5-29e88c3e3a2e None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.664787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d39e419-6c86-440c-958a-29b33d4e0098 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.665004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d39e419-6c86-440c-958a-29b33d4e0098 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.665383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d39e419-6c86-440c-958a-29b33d4e0098 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.666337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5963146d-ed86-4990-b5e5-29e88c3e3a2e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:26.666867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5963146d-ed86-4990-b5e5-29e88c3e3a2e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:26.667832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 315/630] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:26.668926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5963146d-ed86-4990-b5e5-29e88c3e3a2e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:26.669046 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d39e419-6c86-440c-958a-29b33d4e0098 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:26.687058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:26.690901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 316/631] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:26.692732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2395c68-2ba0-450b-9e65-fa63da0ea34f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.693196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2395c68-2ba0-450b-9e65-fa63da0ea34f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.693641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2395c68-2ba0-450b-9e65-fa63da0ea34f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.722560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a2395c68-2ba0-450b-9e65-fa63da0ea34f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:26.724111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a2395c68-2ba0-450b-9e65-fa63da0ea34f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:26.796427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d39e419-6c86-440c-958a-29b33d4e0098 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:26.808071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 317/632] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:26.808864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2395c68-2ba0-450b-9e65-fa63da0ea34f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:26.824847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:26.833549 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410d2f0f-1893-408d-beb4-651772d06943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:26.834390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410d2f0f-1893-408d-beb4-651772d06943 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.834826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410d2f0f-1893-408d-beb4-651772d06943 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.835255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410d2f0f-1893-408d-beb4-651772d06943 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.845983 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410d2f0f-1893-408d-beb4-651772d06943 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:26.847611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410d2f0f-1893-408d-beb4-651772d06943 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-410d2f0f-1893-408d-beb4-651772d06943 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:26.863043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:26.864141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:26.865099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:26.865099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:26.865099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:26.865511 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 316/633] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:26.866293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410d2f0f-1893-408d-beb4-651772d06943 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:26.924740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2395c68-2ba0-450b-9e65-fa63da0ea34f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:26.938016 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:26.962798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0d6ab7-e4cf-48dd-8db0-2d487c13470e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:26.963688 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d6ab7-e4cf-48dd-8db0-2d487c13470e None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:26.963967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d6ab7-e4cf-48dd-8db0-2d487c13470e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:26.964238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d6ab7-e4cf-48dd-8db0-2d487c13470e None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/e542b14838e2466fb59d17e9a20682dd/roles/03dc4e232ede47a699064dda38338428` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:26.979867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0d6ab7-e4cf-48dd-8db0-2d487c13470e None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=e542b14838e2466fb59d17e9a20682dd, role_id=03dc4e232ede47a699064dda38338428)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:26.980626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410d2f0f-1893-408d-beb4-651772d06943 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:26.981370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0d6ab7-e4cf-48dd-8db0-2d487c13470e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:27.000633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 318/634] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:23:26 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/e542b14838e2466fb59d17e9a20682dd/roles/03dc4e232ede47a699064dda38338428 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:23:27.002067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56779f39-c7e1-414b-a17c-1f499606c29b None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.002350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56779f39-c7e1-414b-a17c-1f499606c29b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.002680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56779f39-c7e1-414b-a17c-1f499606c29b None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.004134 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 319/635] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:27.009815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56779f39-c7e1-414b-a17c-1f499606c29b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:27.025099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef91c008-ec89-4b27-9773-f5249869bd98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:27.026215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef91c008-ec89-4b27-9773-f5249869bd98 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.026673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef91c008-ec89-4b27-9773-f5249869bd98 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.027227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef91c008-ec89-4b27-9773-f5249869bd98 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.045449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef91c008-ec89-4b27-9773-f5249869bd98 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ef91c008-ec89-4b27-9773-f5249869bd98 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:27.054260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:27.055992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:27.057055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:27.064003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 317/636] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:27.065535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed620427-baca-4ede-953d-083179fee17f None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.065921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed620427-baca-4ede-953d-083179fee17f None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.066297 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed620427-baca-4ede-953d-083179fee17f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.086356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-ed620427-baca-4ede-953d-083179fee17f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:27.087910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ed620427-baca-4ede-953d-083179fee17f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:27.108979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:27.159994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56779f39-c7e1-414b-a17c-1f499606c29b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:27.170815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 318/637] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:27.171593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed620427-baca-4ede-953d-083179fee17f None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:27.211265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8f01e8-4a22-4297-9df5-3f172dc2227b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:27.213110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8f01e8-4a22-4297-9df5-3f172dc2227b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.213110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8f01e8-4a22-4297-9df5-3f172dc2227b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.213110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8f01e8-4a22-4297-9df5-3f172dc2227b None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.213814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8f01e8-4a22-4297-9df5-3f172dc2227b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:27.214119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8f01e8-4a22-4297-9df5-3f172dc2227b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:27.215506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8f01e8-4a22-4297-9df5-3f172dc2227b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:27.230272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 320/638] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:27.233401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8f01e8-4a22-4297-9df5-3f172dc2227b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:27.244652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:27.315702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed620427-baca-4ede-953d-083179fee17f None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:27.361482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8f01e8-4a22-4297-9df5-3f172dc2227b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:27.366879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e8f52b-4a58-4370-bfa1-0b55c925820d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:27.368519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e8f52b-4a58-4370-bfa1-0b55c925820d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.369346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e8f52b-4a58-4370-bfa1-0b55c925820d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.369779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e8f52b-4a58-4370-bfa1-0b55c925820d None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.371036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e8f52b-4a58-4370-bfa1-0b55c925820d None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:27.371491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e8f52b-4a58-4370-bfa1-0b55c925820d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:27.373418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e8f52b-4a58-4370-bfa1-0b55c925820d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:27.381281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:27.385306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 319/639] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:27.386404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e8f52b-4a58-4370-bfa1-0b55c925820d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:27.401794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f0ecd9-c5cb-4ee3-a3eb-e7b7f507d429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:27.402582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f0ecd9-c5cb-4ee3-a3eb-e7b7f507d429 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.402965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f0ecd9-c5cb-4ee3-a3eb-e7b7f507d429 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.403623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f0ecd9-c5cb-4ee3-a3eb-e7b7f507d429 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.414195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f0ecd9-c5cb-4ee3-a3eb-e7b7f507d429 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-35f0ecd9-c5cb-4ee3-a3eb-e7b7f507d429 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:27.420602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:27.421786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:27.422900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:27.428476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 321/640] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:27.429433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f0ecd9-c5cb-4ee3-a3eb-e7b7f507d429 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:27.517333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e8f52b-4a58-4370-bfa1-0b55c925820d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:27.533652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:27.545973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f0ecd9-c5cb-4ee3-a3eb-e7b7f507d429 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:27.558164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0fc64f-25c8-4ed9-b929-ca8994408874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:27.558791 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fc64f-25c8-4ed9-b929-ca8994408874 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.559068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fc64f-25c8-4ed9-b929-ca8994408874 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.559331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fc64f-25c8-4ed9-b929-ca8994408874 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.566809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0fc64f-25c8-4ed9-b929-ca8994408874 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:27.569456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0fc64f-25c8-4ed9-b929-ca8994408874 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:27.582507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d793a4-fb1a-4e92-b4d2-de831de80dbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:27.583186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d793a4-fb1a-4e92-b4d2-de831de80dbc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.583468 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d793a4-fb1a-4e92-b4d2-de831de80dbc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.583740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d793a4-fb1a-4e92-b4d2-de831de80dbc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-7a0fc64f-25c8-4ed9-b929-ca8994408874 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:27.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:27.585584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:27.586494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:27.586494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:27.586494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:27.586494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:27.586494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:27.586494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:27.586494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:27.586494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:27.591209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 320/641] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:27.592636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0fc64f-25c8-4ed9-b929-ca8994408874 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:27.598391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d793a4-fb1a-4e92-b4d2-de831de80dbc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:27.599861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d793a4-fb1a-4e92-b4d2-de831de80dbc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e0d793a4-fb1a-4e92-b4d2-de831de80dbc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:27.614649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:27.615866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:27.616838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:27.616838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:27.616838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:27.616838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 322/642] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:27.616838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d793a4-fb1a-4e92-b4d2-de831de80dbc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:27.654058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:27.711332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0fc64f-25c8-4ed9-b929-ca8994408874 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:27.730351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d793a4-fb1a-4e92-b4d2-de831de80dbc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:27.746131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362a2693-1a07-4bcd-baed-785deb61f323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:27.746739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362a2693-1a07-4bcd-baed-785deb61f323 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.747019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362a2693-1a07-4bcd-baed-785deb61f323 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.747293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362a2693-1a07-4bcd-baed-785deb61f323 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.747949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362a2693-1a07-4bcd-baed-785deb61f323 None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:27.748251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362a2693-1a07-4bcd-baed-785deb61f323 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:27.749636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362a2693-1a07-4bcd-baed-785deb61f323 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:27.760383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 321/643] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:27.760383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362a2693-1a07-4bcd-baed-785deb61f323 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:27.772004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487bb3e0-b93c-4be6-b68a-261022de796c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:27.772744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487bb3e0-b93c-4be6-b68a-261022de796c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.773100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487bb3e0-b93c-4be6-b68a-261022de796c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.773454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487bb3e0-b93c-4be6-b68a-261022de796c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.785438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487bb3e0-b93c-4be6-b68a-261022de796c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-487bb3e0-b93c-4be6-b68a-261022de796c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:27.791628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:27.792829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:27.793923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:27.795658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 323/644] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:27.796578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487bb3e0-b93c-4be6-b68a-261022de796c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:27.799609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:27.868122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362a2693-1a07-4bcd-baed-785deb61f323 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:27.896036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:27.907555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487bb3e0-b93c-4be6-b68a-261022de796c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:27.907798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98e1168-61c3-4350-a120-f74fd4eac774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:27.908686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98e1168-61c3-4350-a120-f74fd4eac774 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.909175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98e1168-61c3-4350-a120-f74fd4eac774 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.909627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98e1168-61c3-4350-a120-f74fd4eac774 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.910491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98e1168-61c3-4350-a120-f74fd4eac774 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:27.910980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98e1168-61c3-4350-a120-f74fd4eac774 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:27.912599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98e1168-61c3-4350-a120-f74fd4eac774 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:27.927290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 322/645] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:27.928241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98e1168-61c3-4350-a120-f74fd4eac774 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:27.945760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218516a2-3eec-40d2-b03a-0fb29fa82b1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:27.947195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218516a2-3eec-40d2-b03a-0fb29fa82b1e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:27.947480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218516a2-3eec-40d2-b03a-0fb29fa82b1e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:27.947752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218516a2-3eec-40d2-b03a-0fb29fa82b1e None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:27.948458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218516a2-3eec-40d2-b03a-0fb29fa82b1e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:27.948769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218516a2-3eec-40d2-b03a-0fb29fa82b1e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:27.950197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218516a2-3eec-40d2-b03a-0fb29fa82b1e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:27.963312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 324/646] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:27.964274 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218516a2-3eec-40d2-b03a-0fb29fa82b1e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:28.040777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:28.047889 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98e1168-61c3-4350-a120-f74fd4eac774 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:28.075281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218516a2-3eec-40d2-b03a-0fb29fa82b1e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:28.083043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b076431-c839-46e3-a58f-c5450886c868 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:28.083666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b076431-c839-46e3-a58f-c5450886c868 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:28.083941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b076431-c839-46e3-a58f-c5450886c868 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:28.084210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b076431-c839-46e3-a58f-c5450886c868 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:28.094966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b076431-c839-46e3-a58f-c5450886c868 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:28.096406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b076431-c839-46e3-a58f-c5450886c868 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-7b076431-c839-46e3-a58f-c5450886c868 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:28.111498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:28.112770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:28.113683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:28.113683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:28.113683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:28.113683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 323/647] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:28.113683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b076431-c839-46e3-a58f-c5450886c868 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:28.118726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d7c8a2-3de8-4c0c-915e-7f6e75a33660 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:28.119277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7c8a2-3de8-4c0c-915e-7f6e75a33660 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:28.119558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7c8a2-3de8-4c0c-915e-7f6e75a33660 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:28.119832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7c8a2-3de8-4c0c-915e-7f6e75a33660 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:28.120533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d7c8a2-3de8-4c0c-915e-7f6e75a33660 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:28.120842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d7c8a2-3de8-4c0c-915e-7f6e75a33660 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:28.122205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d7c8a2-3de8-4c0c-915e-7f6e75a33660 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:28.130728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 325/648] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:28.131418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d7c8a2-3de8-4c0c-915e-7f6e75a33660 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:28.146861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:28.221304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b076431-c839-46e3-a58f-c5450886c868 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:28.236637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d7c8a2-3de8-4c0c-915e-7f6e75a33660 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:28.259371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e888cc08-809a-4b7d-a5b6-446480dfa646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:28.260459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e888cc08-809a-4b7d-a5b6-446480dfa646 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:28.260939 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e888cc08-809a-4b7d-a5b6-446480dfa646 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:28.261430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e888cc08-809a-4b7d-a5b6-446480dfa646 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:28.262461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e888cc08-809a-4b7d-a5b6-446480dfa646 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=381ad7c239f444da90369dc58779b86c) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:28.262993 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e888cc08-809a-4b7d-a5b6-446480dfa646 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:28.264723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e888cc08-809a-4b7d-a5b6-446480dfa646 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:28.269962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cba64cc-3dd6-4e49-a091-7d87be8122bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:28.270579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cba64cc-3dd6-4e49-a091-7d87be8122bd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:28.270854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cba64cc-3dd6-4e49-a091-7d87be8122bd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:28.271123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cba64cc-3dd6-4e49-a091-7d87be8122bd None admin] PATH_INFO: `/v3/users/glance` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:28.271975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 324/649] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:28 2022] GET /identity/v3/role_assignments?user.id=381ad7c239f444da90369dc58779b86c&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:28.272819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e888cc08-809a-4b7d-a5b6-446480dfa646 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:28.288580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cba64cc-3dd6-4e49-a091-7d87be8122bd None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:28.290439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cba64cc-3dd6-4e49-a091-7d87be8122bd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:28.298490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0cba64cc-3dd6-4e49-a091-7d87be8122bd None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:28.306236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:28.307282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:28.308239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:28.308239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:28.308239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:28.308239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:28.308239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:28.308239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:28.308239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:28.308239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:28.308239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 26 01:23:28.308239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:28.315148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 326/650] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:28 2022] GET /identity/v3/users/glance => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:28.316121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cba64cc-3dd6-4e49-a091-7d87be8122bd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:28.408523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:28.418546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e888cc08-809a-4b7d-a5b6-446480dfa646 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:28.443705 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cba64cc-3dd6-4e49-a091-7d87be8122bd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:28.471820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b394fd9-936d-4ce9-98d9-8ee1e3de6fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:28.473482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b394fd9-936d-4ce9-98d9-8ee1e3de6fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:28.473482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b394fd9-936d-4ce9-98d9-8ee1e3de6fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:28.473482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b394fd9-936d-4ce9-98d9-8ee1e3de6fb9 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:28.473986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b394fd9-936d-4ce9-98d9-8ee1e3de6fb9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:28.474291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b394fd9-936d-4ce9-98d9-8ee1e3de6fb9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:28.475781 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b394fd9-936d-4ce9-98d9-8ee1e3de6fb9 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:28.487769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3b3c13-2b46-43e5-b95e-9ed800fd3883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:28.488421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3b3c13-2b46-43e5-b95e-9ed800fd3883 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:28.488697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3b3c13-2b46-43e5-b95e-9ed800fd3883 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:28.488962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3b3c13-2b46-43e5-b95e-9ed800fd3883 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:28.495045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 325/651] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:28.495694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b394fd9-936d-4ce9-98d9-8ee1e3de6fb9 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:28.503505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3b3c13-2b46-43e5-b95e-9ed800fd3883 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:28.505128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3b3c13-2b46-43e5-b95e-9ed800fd3883 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-dc3b3c13-2b46-43e5-b95e-9ed800fd3883 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:28.522413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:28.523388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:28.525243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:28.525243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:28.525243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:28.525243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 327/652] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:28.525243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3b3c13-2b46-43e5-b95e-9ed800fd3883 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:28.582330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:28.622980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b394fd9-936d-4ce9-98d9-8ee1e3de6fb9 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:28.653740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3b3c13-2b46-43e5-b95e-9ed800fd3883 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:28.662321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5871be78-787a-4922-ab18-1c2efd9173ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:28.662864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5871be78-787a-4922-ab18-1c2efd9173ca None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:28.663140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5871be78-787a-4922-ab18-1c2efd9173ca None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:28.663412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5871be78-787a-4922-ab18-1c2efd9173ca None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:28.664527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5871be78-787a-4922-ab18-1c2efd9173ca None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:28.665554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5871be78-787a-4922-ab18-1c2efd9173ca None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:28.667693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5871be78-787a-4922-ab18-1c2efd9173ca None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:28.682062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:28.691191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 326/653] 10.222.0.25 () {56 vars in 1137 bytes} [Mon Sep 26 01:23:28 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:28.692074 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5871be78-787a-4922-ab18-1c2efd9173ca None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:28.701993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82039725-0756-4b18-ac9b-7b627a64ae86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:28.702706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82039725-0756-4b18-ac9b-7b627a64ae86 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:28.703052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82039725-0756-4b18-ac9b-7b627a64ae86 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:28.703390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82039725-0756-4b18-ac9b-7b627a64ae86 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:28.714619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82039725-0756-4b18-ac9b-7b627a64ae86 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:28.716086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82039725-0756-4b18-ac9b-7b627a64ae86 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-82039725-0756-4b18-ac9b-7b627a64ae86 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:28.728664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:28.729651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 328/654] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:28 2022] GET /identity/v3/users/cinder => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:28.730834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82039725-0756-4b18-ac9b-7b627a64ae86 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:28.813364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5871be78-787a-4922-ab18-1c2efd9173ca None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:28.850619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:28.859209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dc18ee-6336-432b-bda4-476842861347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:28.859641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82039725-0756-4b18-ac9b-7b627a64ae86 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:28.860213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dc18ee-6336-432b-bda4-476842861347 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:28.860650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dc18ee-6336-432b-bda4-476842861347 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:28.861061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dc18ee-6336-432b-bda4-476842861347 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:28.861957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dc18ee-6336-432b-bda4-476842861347 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:28.862533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dc18ee-6336-432b-bda4-476842861347 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:28.863965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dc18ee-6336-432b-bda4-476842861347 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:28.881417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 327/655] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:28.881417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dc18ee-6336-432b-bda4-476842861347 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:28.911642 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a543a26-c691-4be7-9b7f-ae02911381e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:28.913488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a543a26-c691-4be7-9b7f-ae02911381e8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:28.913986 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a543a26-c691-4be7-9b7f-ae02911381e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:28.914686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a543a26-c691-4be7-9b7f-ae02911381e8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:28.988875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:28.990283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dc18ee-6336-432b-bda4-476842861347 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:29.015085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a543a26-c691-4be7-9b7f-ae02911381e8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:29.016680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a543a26-c691-4be7-9b7f-ae02911381e8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:29.034842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cffb482-6772-430c-ae03-a228a54bd9cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:29.035723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cffb482-6772-430c-ae03-a228a54bd9cf None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:29.036178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cffb482-6772-430c-ae03-a228a54bd9cf None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:29.036716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cffb482-6772-430c-ae03-a228a54bd9cf None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:29.037724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cffb482-6772-430c-ae03-a228a54bd9cf None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:29.038202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cffb482-6772-430c-ae03-a228a54bd9cf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:29.039865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cffb482-6772-430c-ae03-a228a54bd9cf None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:29.065343 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 328/656] 10.222.0.25 () {56 vars in 1137 bytes} [Mon Sep 26 01:23:28 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:29.066506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cffb482-6772-430c-ae03-a228a54bd9cf None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:29.121537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:29.140722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 329/657] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:29.141515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a543a26-c691-4be7-9b7f-ae02911381e8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:29.186623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cffb482-6772-430c-ae03-a228a54bd9cf None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:29.226798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aabcd9-b921-4696-b1e4-123cf7d4ef62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:29.228324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aabcd9-b921-4696-b1e4-123cf7d4ef62 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:29.228834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aabcd9-b921-4696-b1e4-123cf7d4ef62 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:29.229305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aabcd9-b921-4696-b1e4-123cf7d4ef62 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:29.237318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:29.238685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7aabcd9-b921-4696-b1e4-123cf7d4ef62 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:29.240578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7aabcd9-b921-4696-b1e4-123cf7d4ef62 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:29.254580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a543a26-c691-4be7-9b7f-ae02911381e8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-f7aabcd9-b921-4696-b1e4-123cf7d4ef62 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:29.255806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:29.256768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:29.257726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:29.257726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:29.257726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:29.257726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:29.257726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:29.257726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:29.257726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:29.257726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:29.257726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 26 01:23:29.257726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:29.260764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 329/658] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:29 2022] GET /identity/v3/users/neutron => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:29.261793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aabcd9-b921-4696-b1e4-123cf7d4ef62 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:29.295748 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85401622-6e8b-47db-a113-9f249929668e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:29.296474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85401622-6e8b-47db-a113-9f249929668e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:29.296749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85401622-6e8b-47db-a113-9f249929668e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:29.297064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85401622-6e8b-47db-a113-9f249929668e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:29.370403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aabcd9-b921-4696-b1e4-123cf7d4ef62 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:29.370785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85401622-6e8b-47db-a113-9f249929668e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:29.372448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85401622-6e8b-47db-a113-9f249929668e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:29.380330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:29.406918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7a8c90-6519-4944-8c9e-f2cbaa71a63c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:29.407548 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7a8c90-6519-4944-8c9e-f2cbaa71a63c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:29.407829 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7a8c90-6519-4944-8c9e-f2cbaa71a63c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:29.408179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7a8c90-6519-4944-8c9e-f2cbaa71a63c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:29.419489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7a8c90-6519-4944-8c9e-f2cbaa71a63c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:29.420960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7a8c90-6519-4944-8c9e-f2cbaa71a63c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-9b7a8c90-6519-4944-8c9e-f2cbaa71a63c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:29.434994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:29.435941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:29.436907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:29.436907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:29.436907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:29.436907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 330/659] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:29.437703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffe183f-e930-4bf6-9abb-ed843114157e None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:29.437703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffe183f-e930-4bf6-9abb-ed843114157e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:29.437873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffe183f-e930-4bf6-9abb-ed843114157e None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:29.439207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 331/660] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:29 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:29.439807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffe183f-e930-4bf6-9abb-ed843114157e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:29.470919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:29.492256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 330/661] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:29.492256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85401622-6e8b-47db-a113-9f249929668e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:29.552646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffe183f-e930-4bf6-9abb-ed843114157e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:29.589003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a276667f-0932-49e5-9e66-79645cd31c2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:29.589623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a276667f-0932-49e5-9e66-79645cd31c2b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:29.589885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a276667f-0932-49e5-9e66-79645cd31c2b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:29.590156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a276667f-0932-49e5-9e66-79645cd31c2b None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:29.590878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a276667f-0932-49e5-9e66-79645cd31c2b None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:29.591183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a276667f-0932-49e5-9e66-79645cd31c2b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:29.592650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a276667f-0932-49e5-9e66-79645cd31c2b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:29.593526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85401622-6e8b-47db-a113-9f249929668e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:29.617565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 332/662] 10.222.0.25 () {56 vars in 1139 bytes} [Mon Sep 26 01:23:29 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:29.617565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81152e62-6846-4875-9397-6cf7b7ebe814 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:29.617565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81152e62-6846-4875-9397-6cf7b7ebe814 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:29.617565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81152e62-6846-4875-9397-6cf7b7ebe814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:29.629924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dad93d4-cf59-4c90-9d95-af14dddcc1b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:29.630521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dad93d4-cf59-4c90-9d95-af14dddcc1b6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:29.630798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dad93d4-cf59-4c90-9d95-af14dddcc1b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:29.631065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dad93d4-cf59-4c90-9d95-af14dddcc1b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:29.631697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dad93d4-cf59-4c90-9d95-af14dddcc1b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:29.631993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dad93d4-cf59-4c90-9d95-af14dddcc1b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:29.633949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dad93d4-cf59-4c90-9d95-af14dddcc1b6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:29.637497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-81152e62-6846-4875-9397-6cf7b7ebe814 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:29.638950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-81152e62-6846-4875-9397-6cf7b7ebe814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:29.645474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 331/663] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:29.646289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dad93d4-cf59-4c90-9d95-af14dddcc1b6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:29.654819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:29.700077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:29.721037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 333/664] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:29.722132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81152e62-6846-4875-9397-6cf7b7ebe814 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:29.786088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dad93d4-cf59-4c90-9d95-af14dddcc1b6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:29.829229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844edc06-eed0-452e-a989-f6d5d601f416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:29.830097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844edc06-eed0-452e-a989-f6d5d601f416 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:29.830533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844edc06-eed0-452e-a989-f6d5d601f416 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:29.831029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844edc06-eed0-452e-a989-f6d5d601f416 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:29.843025 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844edc06-eed0-452e-a989-f6d5d601f416 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:29.844727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844edc06-eed0-452e-a989-f6d5d601f416 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:29.845237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81152e62-6846-4875-9397-6cf7b7ebe814 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-844edc06-eed0-452e-a989-f6d5d601f416 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:29.860672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:29.861673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:29.862591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:29.862591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:29.862591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:29.870045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 332/665] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:29.871284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844edc06-eed0-452e-a989-f6d5d601f416 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:29.885032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6dd5c7-a8e9-4578-9004-86fa505b35fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:29.885735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6dd5c7-a8e9-4578-9004-86fa505b35fb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:29.886081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6dd5c7-a8e9-4578-9004-86fa505b35fb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:29.886725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6dd5c7-a8e9-4578-9004-86fa505b35fb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:29.907130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:29.963757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6dd5c7-a8e9-4578-9004-86fa505b35fb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:29.965865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6dd5c7-a8e9-4578-9004-86fa505b35fb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:29.979087 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:29.987428 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844edc06-eed0-452e-a989-f6d5d601f416 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.024450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ae7a89-cc4c-46a8-a720-cf9280259eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.025096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ae7a89-cc4c-46a8-a720-cf9280259eda None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.025383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ae7a89-cc4c-46a8-a720-cf9280259eda None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.025664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ae7a89-cc4c-46a8-a720-cf9280259eda None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.032601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ae7a89-cc4c-46a8-a720-cf9280259eda None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:30.034054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ae7a89-cc4c-46a8-a720-cf9280259eda None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a4ae7a89-cc4c-46a8-a720-cf9280259eda None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:30.050196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:30.051110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:30.052017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:30.052017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:30.052017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:30.052017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:30.052017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:30.052017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:30.052017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:30.052017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:30.052017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 333/666] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:30.053460 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ae7a89-cc4c-46a8-a720-cf9280259eda None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.087696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 334/667] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:30.089546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6dd5c7-a8e9-4578-9004-86fa505b35fb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.146802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:30.166457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ae7a89-cc4c-46a8-a720-cf9280259eda None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.200194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4e1ec1-77ff-46cd-9b51-ecd724fb3c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.200830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4e1ec1-77ff-46cd-9b51-ecd724fb3c39 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.201117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4e1ec1-77ff-46cd-9b51-ecd724fb3c39 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.201394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4e1ec1-77ff-46cd-9b51-ecd724fb3c39 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.202034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4e1ec1-77ff-46cd-9b51-ecd724fb3c39 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:30.202344 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4e1ec1-77ff-46cd-9b51-ecd724fb3c39 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:30.203751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4e1ec1-77ff-46cd-9b51-ecd724fb3c39 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:30.204711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6dd5c7-a8e9-4578-9004-86fa505b35fb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.216406 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 334/668] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:30.217160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4e1ec1-77ff-46cd-9b51-ecd724fb3c39 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.236836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:30.249777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b104f36-af0b-4d25-a7c0-a759a1023f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.250495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b104f36-af0b-4d25-a7c0-a759a1023f1d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.250779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b104f36-af0b-4d25-a7c0-a759a1023f1d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.251058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b104f36-af0b-4d25-a7c0-a759a1023f1d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.262145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b104f36-af0b-4d25-a7c0-a759a1023f1d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-1b104f36-af0b-4d25-a7c0-a759a1023f1d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:30.266975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:30.267895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 335/669] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:30.269025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b104f36-af0b-4d25-a7c0-a759a1023f1d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.329516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4e1ec1-77ff-46cd-9b51-ecd724fb3c39 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.366247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a1a497-e612-45e9-b722-736ede26a61d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.366853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a1a497-e612-45e9-b722-736ede26a61d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.367132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a1a497-e612-45e9-b722-736ede26a61d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.367403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a1a497-e612-45e9-b722-736ede26a61d None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.368728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a1a497-e612-45e9-b722-736ede26a61d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:30.369035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a1a497-e612-45e9-b722-736ede26a61d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:30.370470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a1a497-e612-45e9-b722-736ede26a61d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:30.385442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b104f36-af0b-4d25-a7c0-a759a1023f1d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.387336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 335/670] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:30.388066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a1a497-e612-45e9-b722-736ede26a61d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.401942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:30.429376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365af6f-380a-42de-871f-57679c01a2a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.430011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365af6f-380a-42de-871f-57679c01a2a8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.430322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365af6f-380a-42de-871f-57679c01a2a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.430596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365af6f-380a-42de-871f-57679c01a2a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.440702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365af6f-380a-42de-871f-57679c01a2a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:30.442150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365af6f-380a-42de-871f-57679c01a2a8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-f365af6f-380a-42de-871f-57679c01a2a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:30.458736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:30.459768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:30.460707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:30.460707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:30.460707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:30.460707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 336/671] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:30.460707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365af6f-380a-42de-871f-57679c01a2a8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.480907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:30.505962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a1a497-e612-45e9-b722-736ede26a61d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.544310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0286c5e8-b052-48b6-ab69-1f9b86b846b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.544997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0286c5e8-b052-48b6-ab69-1f9b86b846b0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.545282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0286c5e8-b052-48b6-ab69-1f9b86b846b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.545537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0286c5e8-b052-48b6-ab69-1f9b86b846b0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.553189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0286c5e8-b052-48b6-ab69-1f9b86b846b0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:30.554657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0286c5e8-b052-48b6-ab69-1f9b86b846b0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0286c5e8-b052-48b6-ab69-1f9b86b846b0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:30.567279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:30.568252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:30.569213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:30.569213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:30.569213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:30.569213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:30.569213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:30.569213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:30.569213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:30.569213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:30.569213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 336/672] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:30.569213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0286c5e8-b052-48b6-ab69-1f9b86b846b0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.585229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365af6f-380a-42de-871f-57679c01a2a8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.623942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9e59a6-cff1-4273-aee1-268099ed0b7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.624625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9e59a6-cff1-4273-aee1-268099ed0b7b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.624876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9e59a6-cff1-4273-aee1-268099ed0b7b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.625145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9e59a6-cff1-4273-aee1-268099ed0b7b None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.625779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9e59a6-cff1-4273-aee1-268099ed0b7b None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:30.626078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9e59a6-cff1-4273-aee1-268099ed0b7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:30.627428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9e59a6-cff1-4273-aee1-268099ed0b7b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:30.636592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 337/673] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:30.637473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9e59a6-cff1-4273-aee1-268099ed0b7b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.663212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:30.683483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0286c5e8-b052-48b6-ab69-1f9b86b846b0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.717263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4752396-4410-42b8-891a-b1cadf51fd54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.718549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4752396-4410-42b8-891a-b1cadf51fd54 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.718832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4752396-4410-42b8-891a-b1cadf51fd54 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.719112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4752396-4410-42b8-891a-b1cadf51fd54 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.719841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4752396-4410-42b8-891a-b1cadf51fd54 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=e07b6890d3004dd78faed48a1ab90086) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:30.720156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4752396-4410-42b8-891a-b1cadf51fd54 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:30.721635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4752396-4410-42b8-891a-b1cadf51fd54 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:30.724419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:30.729941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 337/674] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/role_assignments?user.id=e07b6890d3004dd78faed48a1ab90086&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:30.730724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4752396-4410-42b8-891a-b1cadf51fd54 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.739981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9e59a6-cff1-4273-aee1-268099ed0b7b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.778452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e023b2d-1037-41c6-b173-affc33144b23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.779064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e023b2d-1037-41c6-b173-affc33144b23 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.779344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e023b2d-1037-41c6-b173-affc33144b23 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.779621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e023b2d-1037-41c6-b173-affc33144b23 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.780267 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e023b2d-1037-41c6-b173-affc33144b23 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:30.780613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e023b2d-1037-41c6-b173-affc33144b23 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:30.782033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e023b2d-1037-41c6-b173-affc33144b23 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:30.800761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 338/675] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:30.800761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e023b2d-1037-41c6-b173-affc33144b23 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.869541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4752396-4410-42b8-891a-b1cadf51fd54 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.892788 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:30.910744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749c0d66-af4c-4d27-830b-c5dfe623fe21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.911383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c0d66-af4c-4d27-830b-c5dfe623fe21 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.911667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c0d66-af4c-4d27-830b-c5dfe623fe21 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.911937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c0d66-af4c-4d27-830b-c5dfe623fe21 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.913582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749c0d66-af4c-4d27-830b-c5dfe623fe21 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:30.913582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749c0d66-af4c-4d27-830b-c5dfe623fe21 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:30.914823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749c0d66-af4c-4d27-830b-c5dfe623fe21 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:30.914929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e023b2d-1037-41c6-b173-affc33144b23 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:30.931102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 338/676] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:30.932441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432cf7ff-d250-495a-ae92-add595b99418 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.932717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432cf7ff-d250-495a-ae92-add595b99418 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.932990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432cf7ff-d250-495a-ae92-add595b99418 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.936246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 339/677] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:30.936246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432cf7ff-d250-495a-ae92-add595b99418 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:30.963029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9593e5b-7745-4501-bddb-a25ed422ffa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:30.963646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9593e5b-7745-4501-bddb-a25ed422ffa5 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:30.963922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9593e5b-7745-4501-bddb-a25ed422ffa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:30.964200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9593e5b-7745-4501-bddb-a25ed422ffa5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:30.974649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9593e5b-7745-4501-bddb-a25ed422ffa5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:30.976332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9593e5b-7745-4501-bddb-a25ed422ffa5 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-c9593e5b-7745-4501-bddb-a25ed422ffa5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:30.994955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:30.995933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:30.996887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:30.996887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:30.996887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:30.996887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 339/678] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:30.997605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9593e5b-7745-4501-bddb-a25ed422ffa5 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:31.004318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:31.052306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432cf7ff-d250-495a-ae92-add595b99418 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:31.088201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab20da5d-2c30-4a9f-9678-0b281754af7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:31.088859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab20da5d-2c30-4a9f-9678-0b281754af7e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:31.089138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab20da5d-2c30-4a9f-9678-0b281754af7e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:31.089406 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab20da5d-2c30-4a9f-9678-0b281754af7e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:31.096243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab20da5d-2c30-4a9f-9678-0b281754af7e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:31.097744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab20da5d-2c30-4a9f-9678-0b281754af7e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ab20da5d-2c30-4a9f-9678-0b281754af7e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:31.109523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:31.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 340/679] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b93d3a-989c-4923-b13a-3c92c3cf9e71 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:31.116523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b93d3a-989c-4923-b13a-3c92c3cf9e71 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:31.117155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b93d3a-989c-4923-b13a-3c92c3cf9e71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:31.123823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9593e5b-7745-4501-bddb-a25ed422ffa5 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:31.134080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-33b93d3a-989c-4923-b13a-3c92c3cf9e71 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:31.135375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-33b93d3a-989c-4923-b13a-3c92c3cf9e71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:31.165293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba4733d-a38b-441f-a984-61b50a7293c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:31.165895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba4733d-a38b-441f-a984-61b50a7293c4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:31.166183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba4733d-a38b-441f-a984-61b50a7293c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:31.166455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba4733d-a38b-441f-a984-61b50a7293c4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:31.167204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba4733d-a38b-441f-a984-61b50a7293c4 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=2bb5f76eb08741f3a8c7a3a140e01e24) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:31.167511 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba4733d-a38b-441f-a984-61b50a7293c4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:31.168972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba4733d-a38b-441f-a984-61b50a7293c4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:31.170885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:31.187387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 340/680] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:30 2022] GET /identity/v3/role_assignments?user.id=2bb5f76eb08741f3a8c7a3a140e01e24&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:31.196475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba4733d-a38b-441f-a984-61b50a7293c4 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:31.227831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 341/681] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:31.229023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b93d3a-989c-4923-b13a-3c92c3cf9e71 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:31.316897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba4733d-a38b-441f-a984-61b50a7293c4 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:31.336327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:31.358262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04da27-44cf-445a-a015-0430bf8ec69b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:31.360141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04da27-44cf-445a-a015-0430bf8ec69b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:31.360141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04da27-44cf-445a-a015-0430bf8ec69b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:31.360141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04da27-44cf-445a-a015-0430bf8ec69b None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:31.360402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c04da27-44cf-445a-a015-0430bf8ec69b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:31.360593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c04da27-44cf-445a-a015-0430bf8ec69b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:31.362726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c04da27-44cf-445a-a015-0430bf8ec69b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:31.375525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 341/682] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:31.380798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04da27-44cf-445a-a015-0430bf8ec69b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:31.423877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b93d3a-989c-4923-b13a-3c92c3cf9e71 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:31.453886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:31.466757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2490b6e-ea1b-4a14-9807-e078868fa797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:31.467789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2490b6e-ea1b-4a14-9807-e078868fa797 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:31.468277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2490b6e-ea1b-4a14-9807-e078868fa797 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:31.468912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2490b6e-ea1b-4a14-9807-e078868fa797 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:31.469841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2490b6e-ea1b-4a14-9807-e078868fa797 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:31.470447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2490b6e-ea1b-4a14-9807-e078868fa797 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:31.472209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2490b6e-ea1b-4a14-9807-e078868fa797 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:31.487897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 342/683] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:31.497533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2490b6e-ea1b-4a14-9807-e078868fa797 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:31.507453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04da27-44cf-445a-a015-0430bf8ec69b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:31.557841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20a88d6-2dbb-4d64-9ed0-26e1369f4b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:31.558478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20a88d6-2dbb-4d64-9ed0-26e1369f4b39 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:31.558753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20a88d6-2dbb-4d64-9ed0-26e1369f4b39 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:31.559023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20a88d6-2dbb-4d64-9ed0-26e1369f4b39 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:31.568973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20a88d6-2dbb-4d64-9ed0-26e1369f4b39 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:31.571111 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20a88d6-2dbb-4d64-9ed0-26e1369f4b39 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-e20a88d6-2dbb-4d64-9ed0-26e1369f4b39 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:31.585945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:31.586996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:31.587949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 342/684] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:31 2022] GET /identity/v3/users/swift => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:31.595102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20a88d6-2dbb-4d64-9ed0-26e1369f4b39 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:31.616444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2490b6e-ea1b-4a14-9807-e078868fa797 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:31.629309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:31.656079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88506160-9a3a-4d11-9c41-86eab8b225f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:31.656833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88506160-9a3a-4d11-9c41-86eab8b225f0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:31.657117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88506160-9a3a-4d11-9c41-86eab8b225f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:31.657390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88506160-9a3a-4d11-9c41-86eab8b225f0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:31.658116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88506160-9a3a-4d11-9c41-86eab8b225f0 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=0478ebab37fb4f488eef84f7a241e206) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:31.658522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88506160-9a3a-4d11-9c41-86eab8b225f0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:31.659963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88506160-9a3a-4d11-9c41-86eab8b225f0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:31.672505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 343/685] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:31 2022] GET /identity/v3/role_assignments?user.id=0478ebab37fb4f488eef84f7a241e206&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:31.673595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88506160-9a3a-4d11-9c41-86eab8b225f0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:31.714776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20a88d6-2dbb-4d64-9ed0-26e1369f4b39 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:31.737115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:31.761223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d7a08a-e7f9-4d9f-91a6-d698fc37698c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:31.761854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d7a08a-e7f9-4d9f-91a6-d698fc37698c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:31.762124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d7a08a-e7f9-4d9f-91a6-d698fc37698c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:31.762394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d7a08a-e7f9-4d9f-91a6-d698fc37698c None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:31.763637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d7a08a-e7f9-4d9f-91a6-d698fc37698c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:31.763943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d7a08a-e7f9-4d9f-91a6-d698fc37698c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:31.765411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d7a08a-e7f9-4d9f-91a6-d698fc37698c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:31.810756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 343/686] 10.222.0.25 () {56 vars in 1135 bytes} [Mon Sep 26 01:23:31 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:31.816943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d7a08a-e7f9-4d9f-91a6-d698fc37698c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:31.886981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88506160-9a3a-4d11-9c41-86eab8b225f0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:31.927726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77ee1f6-96f2-4013-91d1-e0e669bcd935 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:31.928446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77ee1f6-96f2-4013-91d1-e0e669bcd935 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:31.928725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77ee1f6-96f2-4013-91d1-e0e669bcd935 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:31.929485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77ee1f6-96f2-4013-91d1-e0e669bcd935 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:31.952782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77ee1f6-96f2-4013-91d1-e0e669bcd935 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b77ee1f6-96f2-4013-91d1-e0e669bcd935 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:31.960253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:31.961246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:31.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 344/687] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:31.968512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77ee1f6-96f2-4013-91d1-e0e669bcd935 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:31.989045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d7a08a-e7f9-4d9f-91a6-d698fc37698c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:32.007579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:32.028827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6969bc73-a76d-44ec-a065-1916f88eac79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:32.029491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6969bc73-a76d-44ec-a065-1916f88eac79 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:32.029771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6969bc73-a76d-44ec-a065-1916f88eac79 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:32.030042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6969bc73-a76d-44ec-a065-1916f88eac79 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:32.083550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:32.112894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77ee1f6-96f2-4013-91d1-e0e669bcd935 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:32.115040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6969bc73-a76d-44ec-a065-1916f88eac79 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:32.116759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6969bc73-a76d-44ec-a065-1916f88eac79 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:32.160648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418f73d-fc99-446d-80eb-267dad5c50fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:32.161610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418f73d-fc99-446d-80eb-267dad5c50fb None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:32.162105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418f73d-fc99-446d-80eb-267dad5c50fb None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:32.162590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418f73d-fc99-446d-80eb-267dad5c50fb None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:32.163458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7418f73d-fc99-446d-80eb-267dad5c50fb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:32.163966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7418f73d-fc99-446d-80eb-267dad5c50fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:32.165816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7418f73d-fc99-446d-80eb-267dad5c50fb None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:32.196009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 345/688] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:32.201572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418f73d-fc99-446d-80eb-267dad5c50fb None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:32.248065 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 344/689] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 26 01:23:32.253487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6969bc73-a76d-44ec-a065-1916f88eac79 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:32.369333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:32.402156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:32.412157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418f73d-fc99-446d-80eb-267dad5c50fb None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:32.419553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6969bc73-a76d-44ec-a065-1916f88eac79 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:32.480975 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b42bd3-5d40-4ea3-ab67-4728cb0a3a70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:32.481604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b42bd3-5d40-4ea3-ab67-4728cb0a3a70 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:32.481880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b42bd3-5d40-4ea3-ab67-4728cb0a3a70 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:32.482150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b42bd3-5d40-4ea3-ab67-4728cb0a3a70 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:32.487389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48782ded-7064-40a4-9a90-d38d42ed808f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:32.488275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48782ded-7064-40a4-9a90-d38d42ed808f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:32.488800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48782ded-7064-40a4-9a90-d38d42ed808f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:32.489294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48782ded-7064-40a4-9a90-d38d42ed808f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:32.500999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48782ded-7064-40a4-9a90-d38d42ed808f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:32.502790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48782ded-7064-40a4-9a90-d38d42ed808f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:32.506236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b42bd3-5d40-4ea3-ab67-4728cb0a3a70 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:32.507682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b42bd3-5d40-4ea3-ab67-4728cb0a3a70 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-48782ded-7064-40a4-9a90-d38d42ed808f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:32.520249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:32.521373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:32.522279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:32.522279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:32.522279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:32.524889 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 345/690] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:32.530238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48782ded-7064-40a4-9a90-d38d42ed808f None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-13b42bd3-5d40-4ea3-ab67-4728cb0a3a70 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:32.542687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:32.543725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:32.544747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:32.544747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:32.544747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:32.544747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 346/691] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 342 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:32.553342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b42bd3-5d40-4ea3-ab67-4728cb0a3a70 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:32.648071 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48782ded-7064-40a4-9a90-d38d42ed808f None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:32.677612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:32.687668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73973c45-6238-4fc8-b07d-a042c18e630d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:32.688643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73973c45-6238-4fc8-b07d-a042c18e630d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:32.689153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73973c45-6238-4fc8-b07d-a042c18e630d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:32.689631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73973c45-6238-4fc8-b07d-a042c18e630d None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:32.690501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73973c45-6238-4fc8-b07d-a042c18e630d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:32.691008 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73973c45-6238-4fc8-b07d-a042c18e630d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:32.692633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73973c45-6238-4fc8-b07d-a042c18e630d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:32.707825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 346/692] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:32.713419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73973c45-6238-4fc8-b07d-a042c18e630d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:32.762599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b42bd3-5d40-4ea3-ab67-4728cb0a3a70 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:32.836207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73973c45-6238-4fc8-b07d-a042c18e630d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:32.854298 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cdbfe2-059d-482b-b2c4-717a6ecb1bf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:32.854910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cdbfe2-059d-482b-b2c4-717a6ecb1bf1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:32.855235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cdbfe2-059d-482b-b2c4-717a6ecb1bf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:32.855434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cdbfe2-059d-482b-b2c4-717a6ecb1bf1 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:32.856085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cdbfe2-059d-482b-b2c4-717a6ecb1bf1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:32.856401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cdbfe2-059d-482b-b2c4-717a6ecb1bf1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:32.857780 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cdbfe2-059d-482b-b2c4-717a6ecb1bf1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:32.877099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7011e95-7b00-4554-994f-61a3c96f881a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:32.878093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7011e95-7b00-4554-994f-61a3c96f881a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:32.878618 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7011e95-7b00-4554-994f-61a3c96f881a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:32.879102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7011e95-7b00-4554-994f-61a3c96f881a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:32.881789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 347/693] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:32.883854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cdbfe2-059d-482b-b2c4-717a6ecb1bf1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:32.887476 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7011e95-7b00-4554-994f-61a3c96f881a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:32.889313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7011e95-7b00-4554-994f-61a3c96f881a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-b7011e95-7b00-4554-994f-61a3c96f881a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:32.902152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:32.903123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:32.904024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:32.904024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:32.904024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:32.904024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:32.904024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:32.904024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:32.904024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:32.904024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:32.906417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 347/694] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:32.917413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7011e95-7b00-4554-994f-61a3c96f881a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:32.920716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:32.981022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:33.026658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7011e95-7b00-4554-994f-61a3c96f881a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:33.085143 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1965bc-82d0-4101-9f3e-ff162eba7fb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:33.089160 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1965bc-82d0-4101-9f3e-ff162eba7fb6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:33.089356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1965bc-82d0-4101-9f3e-ff162eba7fb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:33.089695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1965bc-82d0-4101-9f3e-ff162eba7fb6 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:33.091340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1965bc-82d0-4101-9f3e-ff162eba7fb6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:33.091931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1965bc-82d0-4101-9f3e-ff162eba7fb6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:33.095749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1965bc-82d0-4101-9f3e-ff162eba7fb6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:33.106028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cdbfe2-059d-482b-b2c4-717a6ecb1bf1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:33.122757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 348/695] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:33.126128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1965bc-82d0-4101-9f3e-ff162eba7fb6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:33.167474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8381402-e92b-42f7-87a3-f674f8807e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:33.168110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8381402-e92b-42f7-87a3-f674f8807e54 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:33.169027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8381402-e92b-42f7-87a3-f674f8807e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:33.169311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8381402-e92b-42f7-87a3-f674f8807e54 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:33.187550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8381402-e92b-42f7-87a3-f674f8807e54 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:33.189119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8381402-e92b-42f7-87a3-f674f8807e54 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b8381402-e92b-42f7-87a3-f674f8807e54 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:33.210302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:33.211280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:33.212598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 348/696] 10.222.0.25 () {56 vars in 1093 bytes} [Mon Sep 26 01:23:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 328 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:33.220008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8381402-e92b-42f7-87a3-f674f8807e54 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:33.253614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1965bc-82d0-4101-9f3e-ff162eba7fb6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:33.264120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:33.295877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ed1ce1-fbf0-47fa-9ad7-00311f29714b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:33.296293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ed1ce1-fbf0-47fa-9ad7-00311f29714b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:33.296606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ed1ce1-fbf0-47fa-9ad7-00311f29714b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:33.296881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ed1ce1-fbf0-47fa-9ad7-00311f29714b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:33.297602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ed1ce1-fbf0-47fa-9ad7-00311f29714b None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=e542b14838e2466fb59d17e9a20682dd) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:33.297907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ed1ce1-fbf0-47fa-9ad7-00311f29714b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:33.299450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ed1ce1-fbf0-47fa-9ad7-00311f29714b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:33.306816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 349/697] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:33 2022] GET /identity/v3/role_assignments?user.id=e542b14838e2466fb59d17e9a20682dd&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:33.312936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fe1096-8c61-40cd-a3c1-0837813ce544 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:33.313251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fe1096-8c61-40cd-a3c1-0837813ce544 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:33.313533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fe1096-8c61-40cd-a3c1-0837813ce544 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:33.315324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 350/698] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:33 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:33.320871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d054b4-a828-4410-b27e-790fc3682641 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:33.321229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d054b4-a828-4410-b27e-790fc3682641 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:33.321517 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d054b4-a828-4410-b27e-790fc3682641 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:33.328719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:33.351502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-f2d054b4-a828-4410-b27e-790fc3682641 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:33.353593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f2d054b4-a828-4410-b27e-790fc3682641 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:33.405919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8381402-e92b-42f7-87a3-f674f8807e54 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:33.445320 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 351/699] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:33.477662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a8eba8-4a0a-4b10-bc19-6308503f9ac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:33.478329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a8eba8-4a0a-4b10-bc19-6308503f9ac0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:33.478611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a8eba8-4a0a-4b10-bc19-6308503f9ac0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:33.478884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a8eba8-4a0a-4b10-bc19-6308503f9ac0 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:33.479622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a8eba8-4a0a-4b10-bc19-6308503f9ac0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:33.479931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a8eba8-4a0a-4b10-bc19-6308503f9ac0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:33.481520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a8eba8-4a0a-4b10-bc19-6308503f9ac0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:33.508445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 349/700] 10.222.0.25 () {56 vars in 1133 bytes} [Mon Sep 26 01:23:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:33.514355 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d054b4-a828-4410-b27e-790fc3682641 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:33.642674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d054b4-a828-4410-b27e-790fc3682641 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:33.644245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:33.678775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b25271-1fd0-4d56-b92e-2dce9c25334b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:33.679388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b25271-1fd0-4d56-b92e-2dce9c25334b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:33.679667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b25271-1fd0-4d56-b92e-2dce9c25334b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:33.679937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b25271-1fd0-4d56-b92e-2dce9c25334b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:33.755403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b25271-1fd0-4d56-b92e-2dce9c25334b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:33.756921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b25271-1fd0-4d56-b92e-2dce9c25334b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:33.779197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a8eba8-4a0a-4b10-bc19-6308503f9ac0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:33.886707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 352/701] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:33.888268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebdcc83-a2a8-4f4e-ac39-693c9ee47952 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:33.888585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebdcc83-a2a8-4f4e-ac39-693c9ee47952 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:33.888859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebdcc83-a2a8-4f4e-ac39-693c9ee47952 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:33.891870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 353/702] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:33.897415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ae7354-5562-4451-999f-cc3f35678f94 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:33.897562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ae7354-5562-4451-999f-cc3f35678f94 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:33.897907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ae7354-5562-4451-999f-cc3f35678f94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:33.904598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:33.912432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a8eba8-4a0a-4b10-bc19-6308503f9ac0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:33.921860 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-34ae7354-5562-4451-999f-cc3f35678f94 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:33.923166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-34ae7354-5562-4451-999f-cc3f35678f94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:33.945668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:33.954027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb1f1da-97b8-4679-b3e9-85c6d2112000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:33.954667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb1f1da-97b8-4679-b3e9-85c6d2112000 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:33.954942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb1f1da-97b8-4679-b3e9-85c6d2112000 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:33.955232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb1f1da-97b8-4679-b3e9-85c6d2112000 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:33.966376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb1f1da-97b8-4679-b3e9-85c6d2112000 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:33.968029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb1f1da-97b8-4679-b3e9-85c6d2112000 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-acb1f1da-97b8-4679-b3e9-85c6d2112000 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:33.984424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:33.985690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:33.986817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:33.986817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:33.986817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:33.990564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 350/703] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:33.991416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb1f1da-97b8-4679-b3e9-85c6d2112000 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:34.016382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 354/704] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:23:34.016382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ae7354-5562-4451-999f-cc3f35678f94 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:34.167733 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb1f1da-97b8-4679-b3e9-85c6d2112000 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:34.203761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ae7354-5562-4451-999f-cc3f35678f94 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:34.234752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd39465-65a4-4627-b3c7-913d23c03180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:34.235385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd39465-65a4-4627-b3c7-913d23c03180 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:34.235660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd39465-65a4-4627-b3c7-913d23c03180 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:34.235929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd39465-65a4-4627-b3c7-913d23c03180 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:34.259234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd39465-65a4-4627-b3c7-913d23c03180 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:34.260753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd39465-65a4-4627-b3c7-913d23c03180 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:34.267850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:34.275826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4774e5-732e-4ff5-8118-d907f0d25e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:34.277173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4774e5-732e-4ff5-8118-d907f0d25e47 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:34.277686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4774e5-732e-4ff5-8118-d907f0d25e47 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:34.278298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4774e5-732e-4ff5-8118-d907f0d25e47 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:34.279153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4774e5-732e-4ff5-8118-d907f0d25e47 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:34.279671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4774e5-732e-4ff5-8118-d907f0d25e47 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:34.281319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4774e5-732e-4ff5-8118-d907f0d25e47 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ffd39465-65a4-4627-b3c7-913d23c03180 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:34.289319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:34.290524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:34.295682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:34.295682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:34.295682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:34.295682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 351/705] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:34.297846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd39465-65a4-4627-b3c7-913d23c03180 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:34.310167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 355/706] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:34.311643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4774e5-732e-4ff5-8118-d907f0d25e47 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:34.349539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:34.432363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4774e5-732e-4ff5-8118-d907f0d25e47 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:34.446789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd39465-65a4-4627-b3c7-913d23c03180 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:34.476733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001569d3-e9ff-48bd-a006-a9f06089acd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:34.478342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001569d3-e9ff-48bd-a006-a9f06089acd8 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:34.478974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001569d3-e9ff-48bd-a006-a9f06089acd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:34.479472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001569d3-e9ff-48bd-a006-a9f06089acd8 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:34.481045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001569d3-e9ff-48bd-a006-a9f06089acd8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:34.483085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001569d3-e9ff-48bd-a006-a9f06089acd8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:34.488414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b3f088-a8f7-4e96-96d5-209dc72fbd64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:34.489012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b3f088-a8f7-4e96-96d5-209dc72fbd64 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:34.489290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b3f088-a8f7-4e96-96d5-209dc72fbd64 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:34.489555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b3f088-a8f7-4e96-96d5-209dc72fbd64 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:34.490194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b3f088-a8f7-4e96-96d5-209dc72fbd64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:34.490493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b3f088-a8f7-4e96-96d5-209dc72fbd64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:34.491862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b3f088-a8f7-4e96-96d5-209dc72fbd64 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:34.508285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 352/707] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:34.511267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b3f088-a8f7-4e96-96d5-209dc72fbd64 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:34.516720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 356/708] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:23:34 2022] POST /identity/v3/users => generated 254 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:34.517828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001569d3-e9ff-48bd-a006-a9f06089acd8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:34.591046 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:34.644306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:34.661045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b3f088-a8f7-4e96-96d5-209dc72fbd64 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:34.662327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001569d3-e9ff-48bd-a006-a9f06089acd8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:34.710632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680f456-e078-46f7-919c-48569c6be0b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:34.711436 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680f456-e078-46f7-919c-48569c6be0b2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:34.711789 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680f456-e078-46f7-919c-48569c6be0b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:34.712428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680f456-e078-46f7-919c-48569c6be0b2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:34.713723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a14814c-b24a-430a-9295-6a90714ba316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:34.715116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a14814c-b24a-430a-9295-6a90714ba316 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:34.716178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a14814c-b24a-430a-9295-6a90714ba316 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:34.716728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a14814c-b24a-430a-9295-6a90714ba316 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:34.717619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a14814c-b24a-430a-9295-6a90714ba316 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:34.719264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a14814c-b24a-430a-9295-6a90714ba316 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:34.720974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9680f456-e078-46f7-919c-48569c6be0b2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:34.722732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9680f456-e078-46f7-919c-48569c6be0b2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-9680f456-e078-46f7-919c-48569c6be0b2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:34.737021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:34.737958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:34.739196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:34.739196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:34.739196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:34.739196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:34.739196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:34.739196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:34.739196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:34.739196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:34.739196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 357/709] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:34.739805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f7e950-72e4-4ee9-af7d-e1d24c14a5e5 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:34.739888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f7e950-72e4-4ee9-af7d-e1d24c14a5e5 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:34.740148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f7e950-72e4-4ee9-af7d-e1d24c14a5e5 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4a14814c-b24a-430a-9295-6a90714ba316 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:34.742130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:34.744124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 26 01:23:34.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:34.750225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 358/710] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:34 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:34.750225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f7e950-72e4-4ee9-af7d-e1d24c14a5e5 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:34.751905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 353/711] 10.222.0.25 () {56 vars in 1129 bytes} [Mon Sep 26 01:23:34 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:34.753717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f403b304-9d19-4cb7-a027-bf7cabb49760 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:34.754002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f403b304-9d19-4cb7-a027-bf7cabb49760 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:34.754277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f403b304-9d19-4cb7-a027-bf7cabb49760 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:34.773289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f403b304-9d19-4cb7-a027-bf7cabb49760 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:34.774660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f403b304-9d19-4cb7-a027-bf7cabb49760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:34.876526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 354/712] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:23:34.877727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f403b304-9d19-4cb7-a027-bf7cabb49760 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:34.885594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f7e950-72e4-4ee9-af7d-e1d24c14a5e5 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:34.902198 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:34.951051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d885fc98-f315-4f5d-834a-de3ffad5c6ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:34.951822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d885fc98-f315-4f5d-834a-de3ffad5c6ad None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:34.953705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d885fc98-f315-4f5d-834a-de3ffad5c6ad None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:34.954088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d885fc98-f315-4f5d-834a-de3ffad5c6ad None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:34.954895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d885fc98-f315-4f5d-834a-de3ffad5c6ad None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:34.955327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d885fc98-f315-4f5d-834a-de3ffad5c6ad None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:34.958323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d885fc98-f315-4f5d-834a-de3ffad5c6ad None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:34.973004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 359/713] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:34.981306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d885fc98-f315-4f5d-834a-de3ffad5c6ad None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:35.042668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:35.078924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f403b304-9d19-4cb7-a027-bf7cabb49760 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:35.158823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7962d7da-3ccf-47cc-8c3e-d8f8fb04b08e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:35.159475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7962d7da-3ccf-47cc-8c3e-d8f8fb04b08e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:35.159756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7962d7da-3ccf-47cc-8c3e-d8f8fb04b08e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:35.160030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7962d7da-3ccf-47cc-8c3e-d8f8fb04b08e None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:35.160713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7962d7da-3ccf-47cc-8c3e-d8f8fb04b08e None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:35.161023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7962d7da-3ccf-47cc-8c3e-d8f8fb04b08e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:35.162401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7962d7da-3ccf-47cc-8c3e-d8f8fb04b08e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:35.162958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d885fc98-f315-4f5d-834a-de3ffad5c6ad None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:35.174560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 355/714] 10.222.0.25 () {56 vars in 1124 bytes} [Mon Sep 26 01:23:34 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:35.179941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7962d7da-3ccf-47cc-8c3e-d8f8fb04b08e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:35.203351 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b515d-fad6-4c22-902f-2409f4c2c11c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:35.204254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b515d-fad6-4c22-902f-2409f4c2c11c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:35.204757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b515d-fad6-4c22-902f-2409f4c2c11c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:35.205197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b515d-fad6-4c22-902f-2409f4c2c11c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:35.206204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657b515d-fad6-4c22-902f-2409f4c2c11c None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=381ad7c239f444da90369dc58779b86c) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:35.206940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657b515d-fad6-4c22-902f-2409f4c2c11c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:35.208797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657b515d-fad6-4c22-902f-2409f4c2c11c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:35.216645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 360/715] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:34 2022] GET /identity/v3/role_assignments?user.id=381ad7c239f444da90369dc58779b86c&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 259 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:35.217664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b515d-fad6-4c22-902f-2409f4c2c11c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:35.284794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:35.308307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7962d7da-3ccf-47cc-8c3e-d8f8fb04b08e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:35.373703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b515d-fad6-4c22-902f-2409f4c2c11c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:35.378797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b686ea03-9661-48bf-8e94-296ab07e3ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:35.379722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b686ea03-9661-48bf-8e94-296ab07e3ad4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:35.380188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b686ea03-9661-48bf-8e94-296ab07e3ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:35.380668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b686ea03-9661-48bf-8e94-296ab07e3ad4 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:35.381502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b686ea03-9661-48bf-8e94-296ab07e3ad4 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:35.382013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b686ea03-9661-48bf-8e94-296ab07e3ad4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:35.383907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b686ea03-9661-48bf-8e94-296ab07e3ad4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:35.396287 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 356/716] 10.222.0.25 () {56 vars in 1124 bytes} [Mon Sep 26 01:23:35 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:35.428554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f516497-f21c-47b3-97d2-7f314813d93a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:35.429135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f516497-f21c-47b3-97d2-7f314813d93a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:35.429414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f516497-f21c-47b3-97d2-7f314813d93a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:35.429687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f516497-f21c-47b3-97d2-7f314813d93a None admin] PATH_INFO: `/v3/services/network` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:35.430863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f516497-f21c-47b3-97d2-7f314813d93a None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:35.432307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f516497-f21c-47b3-97d2-7f314813d93a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-2f516497-f21c-47b3-97d2-7f314813d93a None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:35.444000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:35.445122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:35.446513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 361/717] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:35 2022] GET /identity/v3/services/network => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:35.451969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b686ea03-9661-48bf-8e94-296ab07e3ad4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:35.472625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:35.584768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b686ea03-9661-48bf-8e94-296ab07e3ad4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:35.635064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a025eabb-e684-4298-b5d2-0befccf03626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:35.636044 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a025eabb-e684-4298-b5d2-0befccf03626 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:35.636556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a025eabb-e684-4298-b5d2-0befccf03626 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:35.637070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a025eabb-e684-4298-b5d2-0befccf03626 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:35.637927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a025eabb-e684-4298-b5d2-0befccf03626 None admin] RBAC: Adding query filter params (name=network) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:35.638474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a025eabb-e684-4298-b5d2-0befccf03626 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:35.640111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a025eabb-e684-4298-b5d2-0befccf03626 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:35.645755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 357/718] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:23:35 2022] GET /identity/v3/services?name=network => generated 124 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:35.652582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f516497-f21c-47b3-97d2-7f314813d93a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:35.770922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:35.776772 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f516497-f21c-47b3-97d2-7f314813d93a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:35.812679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6af523-6ceb-4226-95c2-f3d61822723f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:35.813294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6af523-6ceb-4226-95c2-f3d61822723f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:35.813574 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6af523-6ceb-4226-95c2-f3d61822723f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:35.813844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6af523-6ceb-4226-95c2-f3d61822723f None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:35.814528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6af523-6ceb-4226-95c2-f3d61822723f None admin] RBAC: Adding query filter params (type=network) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:35.814774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6af523-6ceb-4226-95c2-f3d61822723f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:35.816207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6af523-6ceb-4226-95c2-f3d61822723f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:35.822683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 362/719] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:23:35 2022] GET /identity/v3/services?type=network => generated 124 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:36.252140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb81ec0-b8d6-42e8-bfa3-2a8f0fe52b7a None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:36.252744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb81ec0-b8d6-42e8-bfa3-2a8f0fe52b7a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:36.252744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb81ec0-b8d6-42e8-bfa3-2a8f0fe52b7a None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:36.254437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 358/720] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:36.260604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bbb41-335a-4b67-82f4-b1a74144b144 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:36.261062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bbb41-335a-4b67-82f4-b1a74144b144 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:36.261167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bbb41-335a-4b67-82f4-b1a74144b144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:36.280554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-924bbb41-335a-4b67-82f4-b1a74144b144 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:36.281748 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-924bbb41-335a-4b67-82f4-b1a74144b144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:36.366735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 363/721] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 26 01:23:36.819116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb81ec0-b8d6-42e8-bfa3-2a8f0fe52b7a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:36.911319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:36.937679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb81ec0-b8d6-42e8-bfa3-2a8f0fe52b7a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:36.987075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8710c9c3-634b-4c23-8e7c-957c9b66baf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:36.987708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8710c9c3-634b-4c23-8e7c-957c9b66baf2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:36.987983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8710c9c3-634b-4c23-8e7c-957c9b66baf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:36.988254 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8710c9c3-634b-4c23-8e7c-957c9b66baf2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:36.999212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8710c9c3-634b-4c23-8e7c-957c9b66baf2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-8710c9c3-634b-4c23-8e7c-957c9b66baf2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:37.005337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:37.006330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:37.007206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 359/722] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:37.013521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924bbb41-335a-4b67-82f4-b1a74144b144 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:37.145988 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:37.159261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924bbb41-335a-4b67-82f4-b1a74144b144 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:37.198073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6546a0c-b572-400e-95ed-4960d6be1a00 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:37.198432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6546a0c-b572-400e-95ed-4960d6be1a00 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:37.198611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6546a0c-b572-400e-95ed-4960d6be1a00 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:37.200145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 360/723] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:23:37.205718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007f7682-94df-45c6-a4f4-36c87bbbf2c1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:37.205997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007f7682-94df-45c6-a4f4-36c87bbbf2c1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:37.206266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007f7682-94df-45c6-a4f4-36c87bbbf2c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:37.214905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff5aa2f-0c16-491f-95b3-2770ce530831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:37.215520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff5aa2f-0c16-491f-95b3-2770ce530831 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:37.215795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff5aa2f-0c16-491f-95b3-2770ce530831 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:37.216068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff5aa2f-0c16-491f-95b3-2770ce530831 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:37.216761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff5aa2f-0c16-491f-95b3-2770ce530831 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:37.217063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff5aa2f-0c16-491f-95b3-2770ce530831 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:37.218419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff5aa2f-0c16-491f-95b3-2770ce530831 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:37.228615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 364/724] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:37.234205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff5aa2f-0c16-491f-95b3-2770ce530831 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:37.241374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-007f7682-94df-45c6-a4f4-36c87bbbf2c1 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:37.243179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-007f7682-94df-45c6-a4f4-36c87bbbf2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:37.318888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 361/725] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:37.379244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff5aa2f-0c16-491f-95b3-2770ce530831 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:37.413827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e048fd9-65de-42df-9368-4c48823c952a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:37.414425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e048fd9-65de-42df-9368-4c48823c952a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:37.415251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e048fd9-65de-42df-9368-4c48823c952a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:37.415525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e048fd9-65de-42df-9368-4c48823c952a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:37.426685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e048fd9-65de-42df-9368-4c48823c952a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:37.428134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e048fd9-65de-42df-9368-4c48823c952a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-3e048fd9-65de-42df-9368-4c48823c952a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:37.442576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:37.444827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:37.447049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:37.447049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:37.447049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:37.447049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 365/726] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:37.449662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007f7682-94df-45c6-a4f4-36c87bbbf2c1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:37.537403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:37.588951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007f7682-94df-45c6-a4f4-36c87bbbf2c1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:37.634398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42d1e8c-e3d8-4e09-ba61-a30f6e262861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:37.634999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42d1e8c-e3d8-4e09-ba61-a30f6e262861 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:37.635276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42d1e8c-e3d8-4e09-ba61-a30f6e262861 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:37.635550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42d1e8c-e3d8-4e09-ba61-a30f6e262861 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:37.636189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42d1e8c-e3d8-4e09-ba61-a30f6e262861 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:37.636530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42d1e8c-e3d8-4e09-ba61-a30f6e262861 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:37.637902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42d1e8c-e3d8-4e09-ba61-a30f6e262861 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:37.659693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 362/727] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:37.665592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e048fd9-65de-42df-9368-4c48823c952a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:37.677835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:37.765938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42d1e8c-e3d8-4e09-ba61-a30f6e262861 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:37.880526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e048fd9-65de-42df-9368-4c48823c952a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:37.917447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42d1e8c-e3d8-4e09-ba61-a30f6e262861 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:37.921664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eadc68-5a14-40ad-8682-6f51b1caa330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:37.922584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eadc68-5a14-40ad-8682-6f51b1caa330 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:37.923061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eadc68-5a14-40ad-8682-6f51b1caa330 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:37.923525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eadc68-5a14-40ad-8682-6f51b1caa330 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:37.932686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eadc68-5a14-40ad-8682-6f51b1caa330 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:37.934554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eadc68-5a14-40ad-8682-6f51b1caa330 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:37.946693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-b0eadc68-5a14-40ad-8682-6f51b1caa330 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:37.948335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:37.949332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:37.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:37.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:37.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:37.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:37.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:37.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:37.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:37.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:37.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:37.950324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:37.957538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 366/728] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:37 2022] GET /identity/v3/users/swift => generated 83 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:37.958909 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5201c44f-4775-4a64-be4b-1b7b1a310b22 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:37.959189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5201c44f-4775-4a64-be4b-1b7b1a310b22 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:37.959459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5201c44f-4775-4a64-be4b-1b7b1a310b22 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:37.962236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 367/729] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:37.962710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986ff7ee-ef45-4c95-a056-0d6246a4824e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:37.963418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986ff7ee-ef45-4c95-a056-0d6246a4824e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:37.963541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5201c44f-4775-4a64-be4b-1b7b1a310b22 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:37.963704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986ff7ee-ef45-4c95-a056-0d6246a4824e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:37.963960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986ff7ee-ef45-4c95-a056-0d6246a4824e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:37.978133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986ff7ee-ef45-4c95-a056-0d6246a4824e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-986ff7ee-ef45-4c95-a056-0d6246a4824e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:37.985681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:37.986647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:37.987501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 363/730] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:37.988713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e641c22-2b81-402a-b353-9a4a8efbe251 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:37.992615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e641c22-2b81-402a-b353-9a4a8efbe251 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:37.993796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e641c22-2b81-402a-b353-9a4a8efbe251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.018364 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7e641c22-2b81-402a-b353-9a4a8efbe251 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:38.021066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7e641c22-2b81-402a-b353-9a4a8efbe251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:38.056243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:38.110025 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 364/731] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:38.111244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e641c22-2b81-402a-b353-9a4a8efbe251 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:38.228834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e641c22-2b81-402a-b353-9a4a8efbe251 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:38.249545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:38.270458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb82799c-bcd4-4400-b120-bac0783e8cde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:38.271535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82799c-bcd4-4400-b120-bac0783e8cde None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.272014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82799c-bcd4-4400-b120-bac0783e8cde None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.272536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82799c-bcd4-4400-b120-bac0783e8cde None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.273392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb82799c-bcd4-4400-b120-bac0783e8cde None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:38.273921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb82799c-bcd4-4400-b120-bac0783e8cde None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:38.275917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb82799c-bcd4-4400-b120-bac0783e8cde None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:38.286032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 365/732] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:38.293300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb82799c-bcd4-4400-b120-bac0783e8cde None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:38.346612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5201c44f-4775-4a64-be4b-1b7b1a310b22 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:38.389816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c553021-d47c-4802-aefd-87b4b1b03f30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:38.390440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c553021-d47c-4802-aefd-87b4b1b03f30 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.390751 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c553021-d47c-4802-aefd-87b4b1b03f30 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.391028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c553021-d47c-4802-aefd-87b4b1b03f30 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.391807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c553021-d47c-4802-aefd-87b4b1b03f30 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:38.392119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c553021-d47c-4802-aefd-87b4b1b03f30 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:38.393626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c553021-d47c-4802-aefd-87b4b1b03f30 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:38.412126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 368/733] 10.222.0.25 () {56 vars in 1135 bytes} [Mon Sep 26 01:23:37 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 449 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:38.413241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a10eaa-f24b-4856-9c6b-7c3e0284081d None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.413573 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a10eaa-f24b-4856-9c6b-7c3e0284081d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.413858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a10eaa-f24b-4856-9c6b-7c3e0284081d None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.419707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 369/734] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:38.419707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a10eaa-f24b-4856-9c6b-7c3e0284081d None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:38.424329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb82799c-bcd4-4400-b120-bac0783e8cde None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:38.461952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827a1bdb-768c-4587-923f-54fb16651156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:38.462687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827a1bdb-768c-4587-923f-54fb16651156 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.462975 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827a1bdb-768c-4587-923f-54fb16651156 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.463250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827a1bdb-768c-4587-923f-54fb16651156 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.475157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827a1bdb-768c-4587-923f-54fb16651156 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:38.477495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827a1bdb-768c-4587-923f-54fb16651156 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-827a1bdb-768c-4587-923f-54fb16651156 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:38.495787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:38.496890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:38.502868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:38.502868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:38.502868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:38.502868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 366/735] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:38.502868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d4270-f6f4-4ebf-9eb0-b5f0e431b302 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.502868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d4270-f6f4-4ebf-9eb0-b5f0e431b302 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.502868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d4270-f6f4-4ebf-9eb0-b5f0e431b302 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.520527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-683d4270-f6f4-4ebf-9eb0-b5f0e431b302 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:38.521704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-683d4270-f6f4-4ebf-9eb0-b5f0e431b302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:38.544928 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a10eaa-f24b-4856-9c6b-7c3e0284081d None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:38.546263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:38.572848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:38.587688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ccc6ee-823a-4a97-9417-b868dcc478d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:38.588738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ccc6ee-823a-4a97-9417-b868dcc478d6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.589165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ccc6ee-823a-4a97-9417-b868dcc478d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.589443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ccc6ee-823a-4a97-9417-b868dcc478d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.604366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 367/736] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:38.604366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d4270-f6f4-4ebf-9eb0-b5f0e431b302 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:38.658972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ccc6ee-823a-4a97-9417-b868dcc478d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:38.660579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ccc6ee-823a-4a97-9417-b868dcc478d6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:38.708666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d4270-f6f4-4ebf-9eb0-b5f0e431b302 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:38.745046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e094e221-643e-439e-932f-52e878b3c34c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:38.745672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094e221-643e-439e-932f-52e878b3c34c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.746017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094e221-643e-439e-932f-52e878b3c34c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.746222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094e221-643e-439e-932f-52e878b3c34c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.756220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e094e221-643e-439e-932f-52e878b3c34c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:38.761111 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 370/737] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e094e221-643e-439e-932f-52e878b3c34c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:38.761238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:38.762117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:38.762971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:38.765142 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ccc6ee-823a-4a97-9417-b868dcc478d6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:38.767342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 368/738] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:38.768621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ce9526-4396-47b5-859d-20e925e00cda None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.768906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ce9526-4396-47b5-859d-20e925e00cda None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.769176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ce9526-4396-47b5-859d-20e925e00cda None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.770698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 369/739] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:38.771405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ce9526-4396-47b5-859d-20e925e00cda None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:38.792757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:38.796185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:38.874002 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ccc6ee-823a-4a97-9417-b868dcc478d6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:38.880936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ce9526-4396-47b5-859d-20e925e00cda None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:38.907293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9673409-15fb-42b4-815e-f3f4c77765c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:38.907895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9673409-15fb-42b4-815e-f3f4c77765c9 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.908172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9673409-15fb-42b4-815e-f3f4c77765c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.908470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9673409-15fb-42b4-815e-f3f4c77765c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.909111 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9673409-15fb-42b4-815e-f3f4c77765c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:38.909413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9673409-15fb-42b4-815e-f3f4c77765c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:38.910756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9673409-15fb-42b4-815e-f3f4c77765c9 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:38.915093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3206d98-6756-4430-833f-78899eb22580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:38.915706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3206d98-6756-4430-833f-78899eb22580 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.915978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3206d98-6756-4430-833f-78899eb22580 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.916246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3206d98-6756-4430-833f-78899eb22580 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.927067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 371/740] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:38.927067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9673409-15fb-42b4-815e-f3f4c77765c9 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:38.927303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3206d98-6756-4430-833f-78899eb22580 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:38.928408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3206d98-6756-4430-833f-78899eb22580 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e3206d98-6756-4430-833f-78899eb22580 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:38.943709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:38.944779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:38.946357 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:38.946357 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:38.946357 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:38.946357 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 370/741] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:38.946586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bac048-84e7-4dae-b14a-96226d4e4015 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:38.947361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bac048-84e7-4dae-b14a-96226d4e4015 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:38.947361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bac048-84e7-4dae-b14a-96226d4e4015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:38.964588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-75bac048-84e7-4dae-b14a-96226d4e4015 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:38.965774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-75bac048-84e7-4dae-b14a-96226d4e4015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:39.006356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:39.025300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9673409-15fb-42b4-815e-f3f4c77765c9 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:39.031768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 371/742] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 26 01:23:39.032396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bac048-84e7-4dae-b14a-96226d4e4015 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:39.034534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:39.062151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314d44aa-8217-4493-8d30-21c5b6e0ac39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:39.062765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314d44aa-8217-4493-8d30-21c5b6e0ac39 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:39.063043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314d44aa-8217-4493-8d30-21c5b6e0ac39 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:39.063314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314d44aa-8217-4493-8d30-21c5b6e0ac39 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:39.063977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314d44aa-8217-4493-8d30-21c5b6e0ac39 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:39.064278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314d44aa-8217-4493-8d30-21c5b6e0ac39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:39.066102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314d44aa-8217-4493-8d30-21c5b6e0ac39 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:39.075312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 372/743] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:39.077485 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314d44aa-8217-4493-8d30-21c5b6e0ac39 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:39.146166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bac048-84e7-4dae-b14a-96226d4e4015 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:39.181986 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14710d39-f496-4dc4-93f4-353dedefa59e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:39.182592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14710d39-f496-4dc4-93f4-353dedefa59e None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:39.183086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14710d39-f496-4dc4-93f4-353dedefa59e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:39.183373 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14710d39-f496-4dc4-93f4-353dedefa59e None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:39.183961 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14710d39-f496-4dc4-93f4-353dedefa59e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:39.185604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14710d39-f496-4dc4-93f4-353dedefa59e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:39.191858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314d44aa-8217-4493-8d30-21c5b6e0ac39 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:39.202742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 372/744] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:23:39 2022] POST /identity/v3/services => generated 251 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:39.203888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14710d39-f496-4dc4-93f4-353dedefa59e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:39.261653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af70d4c2-4720-4fc2-a5f0-fa4f93ebe8d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:39.262321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af70d4c2-4720-4fc2-a5f0-fa4f93ebe8d8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:39.262672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af70d4c2-4720-4fc2-a5f0-fa4f93ebe8d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:39.263014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af70d4c2-4720-4fc2-a5f0-fa4f93ebe8d8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:39.273499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af70d4c2-4720-4fc2-a5f0-fa4f93ebe8d8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:39.277160 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af70d4c2-4720-4fc2-a5f0-fa4f93ebe8d8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:39.280659 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-af70d4c2-4720-4fc2-a5f0-fa4f93ebe8d8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:39.293230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:39.294501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:39.295687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:39.295687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:39.295687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:39.295687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:39.295687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:39.295687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:39.295687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:39.295687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:39.295687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:23:39.295687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:39.300525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 373/745] 10.222.0.25 () {56 vars in 1117 bytes} [Mon Sep 26 01:23:39 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:39.301819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af70d4c2-4720-4fc2-a5f0-fa4f93ebe8d8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:39.314888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:39.361174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14710d39-f496-4dc4-93f4-353dedefa59e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:39.402015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c858aa-9f6a-4416-8eb2-430aee981ee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:39.402728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c858aa-9f6a-4416-8eb2-430aee981ee1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:39.402913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c858aa-9f6a-4416-8eb2-430aee981ee1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:39.403156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c858aa-9f6a-4416-8eb2-430aee981ee1 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:39.403864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c858aa-9f6a-4416-8eb2-430aee981ee1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:39.404087 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c858aa-9f6a-4416-8eb2-430aee981ee1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:39.405555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c858aa-9f6a-4416-8eb2-430aee981ee1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:39.419183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 373/746] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:39.420028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c858aa-9f6a-4416-8eb2-430aee981ee1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:39.424300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af70d4c2-4720-4fc2-a5f0-fa4f93ebe8d8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:39.467915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d34db24-0176-4703-af8f-fac9acdc3728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:39.468601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d34db24-0176-4703-af8f-fac9acdc3728 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:39.468968 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d34db24-0176-4703-af8f-fac9acdc3728 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:39.469247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d34db24-0176-4703-af8f-fac9acdc3728 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:39.481503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d34db24-0176-4703-af8f-fac9acdc3728 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:39.483689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d34db24-0176-4703-af8f-fac9acdc3728 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-3d34db24-0176-4703-af8f-fac9acdc3728 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:39.502012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:39.503024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:39.503952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:39.503952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:39.503952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:39.503952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 374/747] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:39.503952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d34db24-0176-4703-af8f-fac9acdc3728 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:39.532883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c858aa-9f6a-4416-8eb2-430aee981ee1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:39.557657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:39.566473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:39.570894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e7bd28-16b2-4399-88a4-435e4b6f1d99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:39.571303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e7bd28-16b2-4399-88a4-435e4b6f1d99 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:39.571582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e7bd28-16b2-4399-88a4-435e4b6f1d99 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:39.571866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e7bd28-16b2-4399-88a4-435e4b6f1d99 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:39.572616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e7bd28-16b2-4399-88a4-435e4b6f1d99 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:39.573025 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e7bd28-16b2-4399-88a4-435e4b6f1d99 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:39.574597 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e7bd28-16b2-4399-88a4-435e4b6f1d99 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:39.593286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 374/748] 10.222.0.25 () {56 vars in 1149 bytes} [Mon Sep 26 01:23:39 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:39.594290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e7bd28-16b2-4399-88a4-435e4b6f1d99 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:39.624953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d34db24-0176-4703-af8f-fac9acdc3728 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:39.662978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1277b6bd-0d58-49a1-a78b-4c95551decaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:39.663960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1277b6bd-0d58-49a1-a78b-4c95551decaf None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:39.664642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1277b6bd-0d58-49a1-a78b-4c95551decaf None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:39.665201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1277b6bd-0d58-49a1-a78b-4c95551decaf None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:39.666202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1277b6bd-0d58-49a1-a78b-4c95551decaf None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:39.667630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1277b6bd-0d58-49a1-a78b-4c95551decaf None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:39.688398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 375/749] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:23:39 2022] POST /identity/v3/services => generated 240 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:39.688398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1277b6bd-0d58-49a1-a78b-4c95551decaf None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:39.708115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e7bd28-16b2-4399-88a4-435e4b6f1d99 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:39.766830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86300d7f-2b87-4adb-aa0d-1d98d8aa023f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:39.767789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86300d7f-2b87-4adb-aa0d-1d98d8aa023f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:39.768217 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86300d7f-2b87-4adb-aa0d-1d98d8aa023f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:39.768979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86300d7f-2b87-4adb-aa0d-1d98d8aa023f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:39.780202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86300d7f-2b87-4adb-aa0d-1d98d8aa023f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:39.783592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86300d7f-2b87-4adb-aa0d-1d98d8aa023f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-86300d7f-2b87-4adb-aa0d-1d98d8aa023f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:39.801471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:39.802478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:39.803416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:39.803416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:39.803416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:39.803416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:39.803416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:39.803416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:39.803416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:39.803416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:39.803416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 375/750] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:39.804669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86300d7f-2b87-4adb-aa0d-1d98d8aa023f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:39.814744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1277b6bd-0d58-49a1-a78b-4c95551decaf None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:39.830425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:39.861984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d034e9-e003-4aad-80ef-a52047ed0e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:39.863058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d034e9-e003-4aad-80ef-a52047ed0e2b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:39.863577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d034e9-e003-4aad-80ef-a52047ed0e2b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:39.864164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d034e9-e003-4aad-80ef-a52047ed0e2b None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:39.865104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d034e9-e003-4aad-80ef-a52047ed0e2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:39.865716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:39.867055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d034e9-e003-4aad-80ef-a52047ed0e2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:39.869892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d034e9-e003-4aad-80ef-a52047ed0e2b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:39.885419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 376/751] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:39.886639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d034e9-e003-4aad-80ef-a52047ed0e2b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:39.947234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86300d7f-2b87-4adb-aa0d-1d98d8aa023f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:39.995745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a203286-e0b1-4a56-9985-7ba9fdb1d9ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:39.996372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a203286-e0b1-4a56-9985-7ba9fdb1d9ec None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:39.996742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a203286-e0b1-4a56-9985-7ba9fdb1d9ec None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:39.997022 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a203286-e0b1-4a56-9985-7ba9fdb1d9ec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:40.008600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d034e9-e003-4aad-80ef-a52047ed0e2b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:40.053177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741dd3ce-0087-4884-a35e-447f13d3c24b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:40.054316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741dd3ce-0087-4884-a35e-447f13d3c24b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:40.054671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741dd3ce-0087-4884-a35e-447f13d3c24b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:40.055165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741dd3ce-0087-4884-a35e-447f13d3c24b None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:40.056621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741dd3ce-0087-4884-a35e-447f13d3c24b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:40.057066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741dd3ce-0087-4884-a35e-447f13d3c24b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:40.058768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741dd3ce-0087-4884-a35e-447f13d3c24b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:40.075102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 377/752] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:40.076454 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741dd3ce-0087-4884-a35e-447f13d3c24b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:40.088846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a203286-e0b1-4a56-9985-7ba9fdb1d9ec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:40.090587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a203286-e0b1-4a56-9985-7ba9fdb1d9ec None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:40.118415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:40.136402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:40.213392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 376/753] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:39 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:23:40.214248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a203286-e0b1-4a56-9985-7ba9fdb1d9ec None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:40.216914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741dd3ce-0087-4884-a35e-447f13d3c24b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:40.266276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee25ed64-f506-4389-8244-ae45afa5ddb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:40.267545 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee25ed64-f506-4389-8244-ae45afa5ddb3 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:40.268546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee25ed64-f506-4389-8244-ae45afa5ddb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:40.269047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee25ed64-f506-4389-8244-ae45afa5ddb3 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:40.279660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee25ed64-f506-4389-8244-ae45afa5ddb3 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:40.282119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee25ed64-f506-4389-8244-ae45afa5ddb3 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ee25ed64-f506-4389-8244-ae45afa5ddb3 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:40.297110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:40.298307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:40.299550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:40.299550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:40.299550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:40.299550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:40.299550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:40.299550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:40.299550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:40.299550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:40.299550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:40.299550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:40.302287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 378/754] 10.222.0.25 () {56 vars in 1093 bytes} [Mon Sep 26 01:23:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:40.303765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee25ed64-f506-4389-8244-ae45afa5ddb3 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:40.328080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a203286-e0b1-4a56-9985-7ba9fdb1d9ec None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:40.375905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58422dbf-a4be-4b84-9817-00c4327ad232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:40.377210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58422dbf-a4be-4b84-9817-00c4327ad232 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:40.377898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58422dbf-a4be-4b84-9817-00c4327ad232 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:40.378318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58422dbf-a4be-4b84-9817-00c4327ad232 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:40.379433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58422dbf-a4be-4b84-9817-00c4327ad232 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=e542b14838e2466fb59d17e9a20682dd) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:40.380295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58422dbf-a4be-4b84-9817-00c4327ad232 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:40.382460 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58422dbf-a4be-4b84-9817-00c4327ad232 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:40.386310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:40.392015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 377/755] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:40 2022] GET /identity/v3/role_assignments?user.id=e542b14838e2466fb59d17e9a20682dd&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:40.393158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58422dbf-a4be-4b84-9817-00c4327ad232 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:40.421580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:40.433610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee25ed64-f506-4389-8244-ae45afa5ddb3 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:40.521137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7043f6a6-cf54-4b8a-825c-39112312e6c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:40.521795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043f6a6-cf54-4b8a-825c-39112312e6c8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:40.522077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043f6a6-cf54-4b8a-825c-39112312e6c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:40.522406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043f6a6-cf54-4b8a-825c-39112312e6c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:40.548135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7043f6a6-cf54-4b8a-825c-39112312e6c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:40.549722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7043f6a6-cf54-4b8a-825c-39112312e6c8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-7043f6a6-cf54-4b8a-825c-39112312e6c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:40.580076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:40.581117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:40.582799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:40.582799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:40.582799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:40.582799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 379/756] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:40.584796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58422dbf-a4be-4b84-9817-00c4327ad232 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:40.589376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7043f6a6-cf54-4b8a-825c-39112312e6c8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:40.630797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63657ed-cdbd-4ccf-947d-300637353179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:40.631456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63657ed-cdbd-4ccf-947d-300637353179 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:40.631743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63657ed-cdbd-4ccf-947d-300637353179 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:40.632018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63657ed-cdbd-4ccf-947d-300637353179 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:40.632784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63657ed-cdbd-4ccf-947d-300637353179 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:40.633677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63657ed-cdbd-4ccf-947d-300637353179 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:40.635359 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63657ed-cdbd-4ccf-947d-300637353179 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:40.656432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 378/757] 10.222.0.25 () {56 vars in 1133 bytes} [Mon Sep 26 01:23:40 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:40.669526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63657ed-cdbd-4ccf-947d-300637353179 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:40.752624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7043f6a6-cf54-4b8a-825c-39112312e6c8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:40.784812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:40.823697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ebfd0-67af-4729-92ee-f3790c72b783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:40.824371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ebfd0-67af-4729-92ee-f3790c72b783 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:40.824649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ebfd0-67af-4729-92ee-f3790c72b783 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:40.824964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337ebfd0-67af-4729-92ee-f3790c72b783 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:40.825615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337ebfd0-67af-4729-92ee-f3790c72b783 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:40.825915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337ebfd0-67af-4729-92ee-f3790c72b783 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:40.827278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337ebfd0-67af-4729-92ee-f3790c72b783 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:40.851593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 380/758] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:40.858568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ebfd0-67af-4729-92ee-f3790c72b783 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:40.863973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:40.873339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63657ed-cdbd-4ccf-947d-300637353179 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:40.936153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226c6213-08a3-49ca-bed0-43ede94ec1ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:40.936812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226c6213-08a3-49ca-bed0-43ede94ec1ed None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:40.937092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226c6213-08a3-49ca-bed0-43ede94ec1ed None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:40.937366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226c6213-08a3-49ca-bed0-43ede94ec1ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:40.955516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226c6213-08a3-49ca-bed0-43ede94ec1ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:40.957306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226c6213-08a3-49ca-bed0-43ede94ec1ed None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-226c6213-08a3-49ca-bed0-43ede94ec1ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:40.975664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:40.976721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:40.978247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:40.978247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:40.978247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:40.978247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 379/759] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:40.984849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226c6213-08a3-49ca-bed0-43ede94ec1ed None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:41.038280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337ebfd0-67af-4729-92ee-f3790c72b783 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:41.106783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a623d78-db74-42b2-95b9-4eb1600344b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:41.107328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a623d78-db74-42b2-95b9-4eb1600344b9 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:41.107607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a623d78-db74-42b2-95b9-4eb1600344b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:41.107876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a623d78-db74-42b2-95b9-4eb1600344b9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:41.125321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a623d78-db74-42b2-95b9-4eb1600344b9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:41.126780 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a623d78-db74-42b2-95b9-4eb1600344b9 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-8a623d78-db74-42b2-95b9-4eb1600344b9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:41.142913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:41.143872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:41.144801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:41.144801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:41.144801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:41.144801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:41.144801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:41.144801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:41.144801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:41.144801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:41.150315 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 381/760] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:41.160237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a623d78-db74-42b2-95b9-4eb1600344b9 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:41.178685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226c6213-08a3-49ca-bed0-43ede94ec1ed None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:41.197273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:41.241112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142c253e-79db-4d26-8b3a-6c481a2740a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:41.241724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142c253e-79db-4d26-8b3a-6c481a2740a3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:41.242003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142c253e-79db-4d26-8b3a-6c481a2740a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:41.242277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142c253e-79db-4d26-8b3a-6c481a2740a3 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:41.243647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142c253e-79db-4d26-8b3a-6c481a2740a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:41.243954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142c253e-79db-4d26-8b3a-6c481a2740a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:41.245358 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142c253e-79db-4d26-8b3a-6c481a2740a3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:41.269132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 380/761] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:41.274447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142c253e-79db-4d26-8b3a-6c481a2740a3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:41.276735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:41.353130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a623d78-db74-42b2-95b9-4eb1600344b9 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:41.421151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142c253e-79db-4d26-8b3a-6c481a2740a3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:41.447123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec23582a-3c30-4aac-a926-3889caac13f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:41.447781 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec23582a-3c30-4aac-a926-3889caac13f2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:41.448010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec23582a-3c30-4aac-a926-3889caac13f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:41.448276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec23582a-3c30-4aac-a926-3889caac13f2 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:41.448996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec23582a-3c30-4aac-a926-3889caac13f2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:41.449297 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec23582a-3c30-4aac-a926-3889caac13f2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:41.450719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec23582a-3c30-4aac-a926-3889caac13f2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:41.466115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1729ced-6b13-4d95-82e3-008633921a33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:41.466767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1729ced-6b13-4d95-82e3-008633921a33 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:41.467046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1729ced-6b13-4d95-82e3-008633921a33 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:41.467315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1729ced-6b13-4d95-82e3-008633921a33 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:41.478436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1729ced-6b13-4d95-82e3-008633921a33 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:41.479939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1729ced-6b13-4d95-82e3-008633921a33 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:41.487871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 382/762] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:41.494396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec23582a-3c30-4aac-a926-3889caac13f2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a1729ced-6b13-4d95-82e3-008633921a33 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:41.499245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:41.500148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:41.501076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:41.501076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:41.501076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:41.501076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:41.501076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:41.501076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:41.501076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:41.501076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:41.501076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 381/763] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:41.507853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1729ced-6b13-4d95-82e3-008633921a33 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:41.564666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:41.658117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1729ced-6b13-4d95-82e3-008633921a33 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:41.658892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec23582a-3c30-4aac-a926-3889caac13f2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:41.685441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:41.697534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ff32b7-8e73-4bf0-b82b-ffc59f0c0dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:41.698147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ff32b7-8e73-4bf0-b82b-ffc59f0c0dc1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:41.698430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ff32b7-8e73-4bf0-b82b-ffc59f0c0dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:41.698705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ff32b7-8e73-4bf0-b82b-ffc59f0c0dc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:41.699430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ff32b7-8e73-4bf0-b82b-ffc59f0c0dc1 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=50a8d0782b6f47d1aec027c6e15018a8) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:41.699740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ff32b7-8e73-4bf0-b82b-ffc59f0c0dc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:41.700287 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a46058-1c3b-43c8-9cc6-c5f0c06b17e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:41.701172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a46058-1c3b-43c8-9cc6-c5f0c06b17e4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:41.701278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ff32b7-8e73-4bf0-b82b-ffc59f0c0dc1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:41.701739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a46058-1c3b-43c8-9cc6-c5f0c06b17e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:41.702206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a46058-1c3b-43c8-9cc6-c5f0c06b17e4 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:41.703186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a46058-1c3b-43c8-9cc6-c5f0c06b17e4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:41.703700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a46058-1c3b-43c8-9cc6-c5f0c06b17e4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:41.705409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a46058-1c3b-43c8-9cc6-c5f0c06b17e4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:41.707524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 383/764] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:41 2022] GET /identity/v3/role_assignments?user.id=50a8d0782b6f47d1aec027c6e15018a8&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 259 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:41.708625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba369fe-a5ca-4289-82ad-31cc04b5d0dc None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:41.708904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba369fe-a5ca-4289-82ad-31cc04b5d0dc None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:41.709190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba369fe-a5ca-4289-82ad-31cc04b5d0dc None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:41.715277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 384/765] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:41.717246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2196bc77-d226-4247-81fe-a52ae2dfd487 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:41.717539 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2196bc77-d226-4247-81fe-a52ae2dfd487 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:41.717811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2196bc77-d226-4247-81fe-a52ae2dfd487 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:41.725112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 382/766] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:41.731702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a46058-1c3b-43c8-9cc6-c5f0c06b17e4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:41.746824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-2196bc77-d226-4247-81fe-a52ae2dfd487 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:41.748085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2196bc77-d226-4247-81fe-a52ae2dfd487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:41.889069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 385/767] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:41 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:23:41.927346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:41.938861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a46058-1c3b-43c8-9cc6-c5f0c06b17e4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:41.987311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621a95c9-314a-497d-9cad-9c1ed0f5aec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:41.987985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621a95c9-314a-497d-9cad-9c1ed0f5aec3 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:41.988269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621a95c9-314a-497d-9cad-9c1ed0f5aec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:41.988641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621a95c9-314a-497d-9cad-9c1ed0f5aec3 None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/381ad7c239f444da90369dc58779b86c/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:41.990747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621a95c9-314a-497d-9cad-9c1ed0f5aec3 None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=381ad7c239f444da90369dc58779b86c, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:41.992242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621a95c9-314a-497d-9cad-9c1ed0f5aec3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:42.022168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 383/768] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:23:41 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/381ad7c239f444da90369dc58779b86c/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:23:42.102756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9466d2-bf5b-42c0-a4c6-adbae87f6f73 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:42.103126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9466d2-bf5b-42c0-a4c6-adbae87f6f73 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:42.103326 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9466d2-bf5b-42c0-a4c6-adbae87f6f73 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:42.105146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 386/769] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:42.111154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd0a32-0dd8-42e2-8262-c3039c237991 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:42.111439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd0a32-0dd8-42e2-8262-c3039c237991 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:42.111780 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd0a32-0dd8-42e2-8262-c3039c237991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:42.140993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-31dd0a32-0dd8-42e2-8262-c3039c237991 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:42.142213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-31dd0a32-0dd8-42e2-8262-c3039c237991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:42.235093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 384/770] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:42 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:23:42.275144 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9466d2-bf5b-42c0-a4c6-adbae87f6f73 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:42.381856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9466d2-bf5b-42c0-a4c6-adbae87f6f73 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:42.401152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:42.418747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc95bfa1-ace7-4489-99ae-f982b1fe769a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:42.419710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc95bfa1-ace7-4489-99ae-f982b1fe769a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:42.420186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc95bfa1-ace7-4489-99ae-f982b1fe769a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:42.420733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc95bfa1-ace7-4489-99ae-f982b1fe769a None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:42.421528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc95bfa1-ace7-4489-99ae-f982b1fe769a None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:42.423129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc95bfa1-ace7-4489-99ae-f982b1fe769a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-dc95bfa1-ace7-4489-99ae-f982b1fe769a None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:42.433599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:42.434610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 26 01:23:42.435432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:42.438291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 387/771] 10.222.0.25 () {56 vars in 1129 bytes} [Mon Sep 26 01:23:42 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:42.444224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd0a32-0dd8-42e2-8262-c3039c237991 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:42.537160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:42.558574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd0a32-0dd8-42e2-8262-c3039c237991 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:42.605081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a78e2c-3093-404b-bd46-368fb7c94d80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:42.606086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a78e2c-3093-404b-bd46-368fb7c94d80 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:42.606559 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a78e2c-3093-404b-bd46-368fb7c94d80 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:42.607019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a78e2c-3093-404b-bd46-368fb7c94d80 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:42.607841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a78e2c-3093-404b-bd46-368fb7c94d80 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:42.608369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a78e2c-3093-404b-bd46-368fb7c94d80 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:42.610020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a78e2c-3093-404b-bd46-368fb7c94d80 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:42.621091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 385/772] 10.222.0.25 () {56 vars in 1124 bytes} [Mon Sep 26 01:23:42 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:42.627002 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc95bfa1-ace7-4489-99ae-f982b1fe769a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:42.696873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a78e2c-3093-404b-bd46-368fb7c94d80 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:42.784710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc95bfa1-ace7-4489-99ae-f982b1fe769a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:42.843676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f048de5d-7cb4-427e-9110-7d4fa9ea47af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:42.844200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f048de5d-7cb4-427e-9110-7d4fa9ea47af None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:42.844531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f048de5d-7cb4-427e-9110-7d4fa9ea47af None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:42.848366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f048de5d-7cb4-427e-9110-7d4fa9ea47af None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:42.848366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f048de5d-7cb4-427e-9110-7d4fa9ea47af None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:42.848366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f048de5d-7cb4-427e-9110-7d4fa9ea47af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:42.848366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f048de5d-7cb4-427e-9110-7d4fa9ea47af None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:42.853381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 388/773] 10.222.0.25 () {56 vars in 1124 bytes} [Mon Sep 26 01:23:42 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:42.859449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f048de5d-7cb4-427e-9110-7d4fa9ea47af None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:42.901125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a78e2c-3093-404b-bd46-368fb7c94d80 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:42.943790 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fade7a40-85fd-40c5-a51b-7d0b97d7f71f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:42.944445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fade7a40-85fd-40c5-a51b-7d0b97d7f71f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:42.944725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fade7a40-85fd-40c5-a51b-7d0b97d7f71f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:42.944995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fade7a40-85fd-40c5-a51b-7d0b97d7f71f None admin] PATH_INFO: `/v3/services/network` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:42.946682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fade7a40-85fd-40c5-a51b-7d0b97d7f71f None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:42.949363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fade7a40-85fd-40c5-a51b-7d0b97d7f71f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:42.953304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-fade7a40-85fd-40c5-a51b-7d0b97d7f71f None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:42.960942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:42.962676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 26 01:23:42.963525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:42.966314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 386/774] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:42 2022] GET /identity/v3/services/network => generated 88 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:42.967702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985b0778-ac54-49f5-9ec9-a664e8b79348 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:42.968182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985b0778-ac54-49f5-9ec9-a664e8b79348 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:42.968635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985b0778-ac54-49f5-9ec9-a664e8b79348 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:42.970252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 387/775] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:42.971905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985b0778-ac54-49f5-9ec9-a664e8b79348 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:42.985252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:43.019331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f048de5d-7cb4-427e-9110-7d4fa9ea47af None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:43.061529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3c9ae4-6b43-4907-9b20-7b203f7f0fc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:43.062146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3c9ae4-6b43-4907-9b20-7b203f7f0fc0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:43.062425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3c9ae4-6b43-4907-9b20-7b203f7f0fc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:43.062692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3c9ae4-6b43-4907-9b20-7b203f7f0fc0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:43.063383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3c9ae4-6b43-4907-9b20-7b203f7f0fc0 None admin] RBAC: Adding query filter params (interface=public, service_id=1e56e297255c4e34be87f3e376350375, region_id=RegionOne) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:43.063689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3c9ae4-6b43-4907-9b20-7b203f7f0fc0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:43.065078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3c9ae4-6b43-4907-9b20-7b203f7f0fc0 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:43.073871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 389/776] 10.222.0.25 () {56 vars in 1251 bytes} [Mon Sep 26 01:23:42 2022] GET /identity/v3/endpoints?service_id=1e56e297255c4e34be87f3e376350375&interface=public®ion_id=RegionOne => generated 194 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:43.075191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb31e94-bc81-4b25-911f-d0af4caa8171 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:43.075478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb31e94-bc81-4b25-911f-d0af4caa8171 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:43.075764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb31e94-bc81-4b25-911f-d0af4caa8171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:43.094796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-acb31e94-bc81-4b25-911f-d0af4caa8171 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:43.095898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-acb31e94-bc81-4b25-911f-d0af4caa8171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:43.105209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985b0778-ac54-49f5-9ec9-a664e8b79348 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:43.153771 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7125fb-9933-4e80-8859-237494c467ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:43.154375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7125fb-9933-4e80-8859-237494c467ff None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:43.154651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7125fb-9933-4e80-8859-237494c467ff None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:43.154921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7125fb-9933-4e80-8859-237494c467ff None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:43.155546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7125fb-9933-4e80-8859-237494c467ff None admin] RBAC: Adding query filter params (name=network) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:43.155846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7125fb-9933-4e80-8859-237494c467ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:43.157288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7125fb-9933-4e80-8859-237494c467ff None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:43.167349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 388/777] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:23:42 2022] GET /identity/v3/services?name=network => generated 124 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:43.167926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7125fb-9933-4e80-8859-237494c467ff None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:43.176602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 390/778] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:43 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:43.177799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb31e94-bc81-4b25-911f-d0af4caa8171 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:43.231346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:43.302879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb31e94-bc81-4b25-911f-d0af4caa8171 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:43.315072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:43.343195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13044213-ab62-4847-9bc2-f9083e980f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:43.344171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13044213-ab62-4847-9bc2-f9083e980f5c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:43.344867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13044213-ab62-4847-9bc2-f9083e980f5c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:43.345395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13044213-ab62-4847-9bc2-f9083e980f5c None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:43.346248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13044213-ab62-4847-9bc2-f9083e980f5c None admin] RBAC: Adding query filter params (type=network) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:43.346753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13044213-ab62-4847-9bc2-f9083e980f5c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:43.348461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13044213-ab62-4847-9bc2-f9083e980f5c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:43.354175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 391/779] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:23:43 2022] GET /identity/v3/services?type=network => generated 350 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:43.360067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13044213-ab62-4847-9bc2-f9083e980f5c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:43.365440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7125fb-9933-4e80-8859-237494c467ff None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:43.427030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25742738-42fb-4590-944a-4e6cb81d042d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:43.427971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25742738-42fb-4590-944a-4e6cb81d042d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:43.428603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25742738-42fb-4590-944a-4e6cb81d042d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:43.429078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25742738-42fb-4590-944a-4e6cb81d042d None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:43.430703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25742738-42fb-4590-944a-4e6cb81d042d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:43.432266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25742738-42fb-4590-944a-4e6cb81d042d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:43.438357 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 389/780] 10.222.0.25 () {56 vars in 1087 bytes} [Mon Sep 26 01:23:43 2022] GET /identity/v3/services => generated 772 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:43.491793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13044213-ab62-4847-9bc2-f9083e980f5c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:43.496874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:43.531831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e21b10-ad99-4d3d-b28f-2439831d0067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:43.532959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e21b10-ad99-4d3d-b28f-2439831d0067 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:43.533440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e21b10-ad99-4d3d-b28f-2439831d0067 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:43.533910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e21b10-ad99-4d3d-b28f-2439831d0067 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:43.534816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e21b10-ad99-4d3d-b28f-2439831d0067 None admin] RBAC: Adding query filter params (interface=public, service_id=dc8259ee4e174fe89e17096a56b23500, region_id=RegionOne) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:43.535344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e21b10-ad99-4d3d-b28f-2439831d0067 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:43.536962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e21b10-ad99-4d3d-b28f-2439831d0067 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:43.544304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 392/781] 10.222.0.25 () {56 vars in 1251 bytes} [Mon Sep 26 01:23:43 2022] GET /identity/v3/endpoints?service_id=dc8259ee4e174fe89e17096a56b23500&interface=public®ion_id=RegionOne => generated 194 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:43.550677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25742738-42fb-4590-944a-4e6cb81d042d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:43.569767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e21b10-ad99-4d3d-b28f-2439831d0067 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:43.740293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25742738-42fb-4590-944a-4e6cb81d042d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:43.748525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e21b10-ad99-4d3d-b28f-2439831d0067 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:43.779755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6185e7-e4f3-42dc-9216-ec7f356fe275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:43.780735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6185e7-e4f3-42dc-9216-ec7f356fe275 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:43.781219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6185e7-e4f3-42dc-9216-ec7f356fe275 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:43.781744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6185e7-e4f3-42dc-9216-ec7f356fe275 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:43.782842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6185e7-e4f3-42dc-9216-ec7f356fe275 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:43.786001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3018a07a-f167-4531-9235-cc11f9592eff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:43.786001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3018a07a-f167-4531-9235-cc11f9592eff None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:43.786001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3018a07a-f167-4531-9235-cc11f9592eff None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:43.786001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3018a07a-f167-4531-9235-cc11f9592eff None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:43.787154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6185e7-e4f3-42dc-9216-ec7f356fe275 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:43.790224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:43.795349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 390/782] 10.222.0.25 () {56 vars in 1087 bytes} [Mon Sep 26 01:23:43 2022] GET /identity/v3/services => generated 772 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:43.797582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3018a07a-f167-4531-9235-cc11f9592eff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-3018a07a-f167-4531-9235-cc11f9592eff None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:43.805149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:43.806148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:43.806995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 393/783] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:43.810573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6185e7-e4f3-42dc-9216-ec7f356fe275 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:43.927610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6185e7-e4f3-42dc-9216-ec7f356fe275 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:43.972111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d869ab0d-4e87-4fbf-a78b-d0cea0fde9c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:43.973172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d869ab0d-4e87-4fbf-a78b-d0cea0fde9c5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:43.973660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d869ab0d-4e87-4fbf-a78b-d0cea0fde9c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:43.974138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d869ab0d-4e87-4fbf-a78b-d0cea0fde9c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:43.975040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d869ab0d-4e87-4fbf-a78b-d0cea0fde9c5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:43.975599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d869ab0d-4e87-4fbf-a78b-d0cea0fde9c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:43.977617 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d869ab0d-4e87-4fbf-a78b-d0cea0fde9c5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:43.987866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 391/784] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:43.994298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3018a07a-f167-4531-9235-cc11f9592eff None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:44.053517 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:44.144391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3018a07a-f167-4531-9235-cc11f9592eff None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:44.186232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fafc05d-2979-4860-a6b0-f23742a2e70e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:44.186871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fafc05d-2979-4860-a6b0-f23742a2e70e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:44.187150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fafc05d-2979-4860-a6b0-f23742a2e70e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:44.187425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fafc05d-2979-4860-a6b0-f23742a2e70e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:44.208600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fafc05d-2979-4860-a6b0-f23742a2e70e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:44.210066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fafc05d-2979-4860-a6b0-f23742a2e70e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-9fafc05d-2979-4860-a6b0-f23742a2e70e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:44.228215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:44.229255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:44.230177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:44.230177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:44.230177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:44.230177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 394/785] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:44.234923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d869ab0d-4e87-4fbf-a78b-d0cea0fde9c5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:44.284562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84abc17a-d498-436f-8d3c-1ec9c89c5a57 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:44.284752 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84abc17a-d498-436f-8d3c-1ec9c89c5a57 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:44.285025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84abc17a-d498-436f-8d3c-1ec9c89c5a57 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:44.286732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 395/786] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:44.292215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcd6371-490a-40f7-83e9-67df9c0d6250 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:44.292519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcd6371-490a-40f7-83e9-67df9c0d6250 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:44.292800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcd6371-490a-40f7-83e9-67df9c0d6250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:44.323975 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:44.330001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-cfcd6371-490a-40f7-83e9-67df9c0d6250 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:44.331611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-cfcd6371-490a-40f7-83e9-67df9c0d6250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:44.373665 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d869ab0d-4e87-4fbf-a78b-d0cea0fde9c5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:44.431150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 396/787] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:44 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:23:44.434039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4854ac-76df-4a5c-955e-d68d5a95492c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:44.434845 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4854ac-76df-4a5c-955e-d68d5a95492c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:44.435322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4854ac-76df-4a5c-955e-d68d5a95492c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:44.435766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4854ac-76df-4a5c-955e-d68d5a95492c None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:44.436613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4854ac-76df-4a5c-955e-d68d5a95492c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:44.437089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4854ac-76df-4a5c-955e-d68d5a95492c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:44.438631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4854ac-76df-4a5c-955e-d68d5a95492c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:44.459796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 392/788] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:44.465814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcd6371-490a-40f7-83e9-67df9c0d6250 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:44.485716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:44.608657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcd6371-490a-40f7-83e9-67df9c0d6250 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:44.669662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbe349f-3989-4505-8822-891ef146deef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:44.670307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbe349f-3989-4505-8822-891ef146deef None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:44.670589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbe349f-3989-4505-8822-891ef146deef None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:44.670867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbe349f-3989-4505-8822-891ef146deef None admin] PATH_INFO: `/v3/users/swift` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:44.681451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbe349f-3989-4505-8822-891ef146deef None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:44.683021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbe349f-3989-4505-8822-891ef146deef None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-fbbe349f-3989-4505-8822-891ef146deef None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:44.697837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:44.698806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:44.699745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 397/789] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:44 2022] GET /identity/v3/users/swift => generated 83 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:44.704808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4854ac-76df-4a5c-955e-d68d5a95492c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:44.713471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc305c2-9564-4458-851f-9cea928b368e None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:44.713471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc305c2-9564-4458-851f-9cea928b368e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:44.713471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc305c2-9564-4458-851f-9cea928b368e None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:44.713790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 398/790] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:23:44.722927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc305c2-9564-4458-851f-9cea928b368e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:44.801437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:44.823444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4854ac-76df-4a5c-955e-d68d5a95492c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:44.842038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc305c2-9564-4458-851f-9cea928b368e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:44.848314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:44.864937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56da7791-435d-4bda-b214-1d26d5640d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:44.866013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56da7791-435d-4bda-b214-1d26d5640d0b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:44.866811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56da7791-435d-4bda-b214-1d26d5640d0b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:44.867854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56da7791-435d-4bda-b214-1d26d5640d0b None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:44.869137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56da7791-435d-4bda-b214-1d26d5640d0b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:44.869605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56da7791-435d-4bda-b214-1d26d5640d0b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:44.872074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56da7791-435d-4bda-b214-1d26d5640d0b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:44.885061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a5b0fa-3bb9-4296-b2c2-b77ff607f8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:44.886561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a5b0fa-3bb9-4296-b2c2-b77ff607f8a0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:44.886842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a5b0fa-3bb9-4296-b2c2-b77ff607f8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:44.887115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a5b0fa-3bb9-4296-b2c2-b77ff607f8a0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:44.896812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 393/791] 10.222.0.25 () {56 vars in 1135 bytes} [Mon Sep 26 01:23:44 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:44.899132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879e795c-123c-49ff-be4e-dbb1abff34f1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:44.899308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a5b0fa-3bb9-4296-b2c2-b77ff607f8a0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:44.899965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879e795c-123c-49ff-be4e-dbb1abff34f1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:44.900486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879e795c-123c-49ff-be4e-dbb1abff34f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-92a5b0fa-3bb9-4296-b2c2-b77ff607f8a0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:44.904314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:44.905358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:44.906358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 399/792] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:44.907222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a5b0fa-3bb9-4296-b2c2-b77ff607f8a0 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:44.934755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-879e795c-123c-49ff-be4e-dbb1abff34f1 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:44.936212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-879e795c-123c-49ff-be4e-dbb1abff34f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:45.032518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 394/793] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:44 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:45.032518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879e795c-123c-49ff-be4e-dbb1abff34f1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:45.058259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a5b0fa-3bb9-4296-b2c2-b77ff607f8a0 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:45.110041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:45.126226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4228af2b-964a-472a-9d17-1fc1b6e94c82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:45.127185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4228af2b-964a-472a-9d17-1fc1b6e94c82 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:45.127661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4228af2b-964a-472a-9d17-1fc1b6e94c82 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:45.128126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4228af2b-964a-472a-9d17-1fc1b6e94c82 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:45.152335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4228af2b-964a-472a-9d17-1fc1b6e94c82 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:45.154136 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4228af2b-964a-472a-9d17-1fc1b6e94c82 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-4228af2b-964a-472a-9d17-1fc1b6e94c82 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:45.183865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:45.184910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:45.185820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:45.185820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:45.185820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:45.197424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 400/794] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:45.202954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4228af2b-964a-472a-9d17-1fc1b6e94c82 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:45.224796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879e795c-123c-49ff-be4e-dbb1abff34f1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:45.244729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:45.267290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4851b688-912a-4249-833f-5fdc7da64dbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:45.268292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4851b688-912a-4249-833f-5fdc7da64dbf None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:45.268815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4851b688-912a-4249-833f-5fdc7da64dbf None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:45.269282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4851b688-912a-4249-833f-5fdc7da64dbf None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:45.270405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4851b688-912a-4249-833f-5fdc7da64dbf None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:45.271144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4851b688-912a-4249-833f-5fdc7da64dbf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:45.272768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4851b688-912a-4249-833f-5fdc7da64dbf None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:45.282867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 395/795] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:45.288726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4851b688-912a-4249-833f-5fdc7da64dbf None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:45.400019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4228af2b-964a-472a-9d17-1fc1b6e94c82 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:45.419506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4851b688-912a-4249-833f-5fdc7da64dbf None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:45.442628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95070e94-003c-42ac-8a76-8241dbe8e6e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:45.443233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95070e94-003c-42ac-8a76-8241dbe8e6e4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:45.443509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95070e94-003c-42ac-8a76-8241dbe8e6e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:45.443778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95070e94-003c-42ac-8a76-8241dbe8e6e4 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:45.444442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95070e94-003c-42ac-8a76-8241dbe8e6e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:45.444744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95070e94-003c-42ac-8a76-8241dbe8e6e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:45.447428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95070e94-003c-42ac-8a76-8241dbe8e6e4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:45.465649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9022584-b165-4ded-b193-d594f9579313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:45.466015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 401/796] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:45.466281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9022584-b165-4ded-b193-d594f9579313 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:45.466560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9022584-b165-4ded-b193-d594f9579313 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:45.466827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9022584-b165-4ded-b193-d594f9579313 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:45.467856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95070e94-003c-42ac-8a76-8241dbe8e6e4 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:45.483037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9022584-b165-4ded-b193-d594f9579313 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:45.484875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9022584-b165-4ded-b193-d594f9579313 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f9022584-b165-4ded-b193-d594f9579313 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:45.500869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:45.501868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:45.502751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:45.502751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:45.502751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:45.509428 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 396/797] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:45.510486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9022584-b165-4ded-b193-d594f9579313 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:45.521302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:45.538961 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:45.599081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95070e94-003c-42ac-8a76-8241dbe8e6e4 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:45.619179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9022584-b165-4ded-b193-d594f9579313 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:45.643585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8a9af4-c246-433a-b727-a5b175e6e9e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:45.645058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8a9af4-c246-433a-b727-a5b175e6e9e8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:45.645481 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8a9af4-c246-433a-b727-a5b175e6e9e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:45.645763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8a9af4-c246-433a-b727-a5b175e6e9e8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:45.658372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff642ed8-2a35-4a0c-877d-62d6b1aa5e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:45.659209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8a9af4-c246-433a-b727-a5b175e6e9e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:45.659486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff642ed8-2a35-4a0c-877d-62d6b1aa5e04 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:45.659971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff642ed8-2a35-4a0c-877d-62d6b1aa5e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:45.660472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff642ed8-2a35-4a0c-877d-62d6b1aa5e04 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-1c8a9af4-c246-433a-b727-a5b175e6e9e8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:45.665130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:45.666287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:45.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 402/798] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:45.667856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8a9af4-c246-433a-b727-a5b175e6e9e8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:45.671582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff642ed8-2a35-4a0c-877d-62d6b1aa5e04 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:45.673576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff642ed8-2a35-4a0c-877d-62d6b1aa5e04 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ff642ed8-2a35-4a0c-877d-62d6b1aa5e04 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:45.691766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:45.692817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:45.693753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:45.693753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:45.693753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:45.693753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:45.693753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:45.693753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:45.693753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:45.693753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:45.693753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 397/799] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:45.694277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff642ed8-2a35-4a0c-877d-62d6b1aa5e04 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:45.787813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8a9af4-c246-433a-b727-a5b175e6e9e8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:45.790225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:45.807752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:45.829221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff642ed8-2a35-4a0c-877d-62d6b1aa5e04 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:45.832142 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1d7494-d588-46c3-b5ae-74c278e1e4d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:45.833315 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1d7494-d588-46c3-b5ae-74c278e1e4d8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:45.833788 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1d7494-d588-46c3-b5ae-74c278e1e4d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:45.834254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1d7494-d588-46c3-b5ae-74c278e1e4d8 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:45.835085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1d7494-d588-46c3-b5ae-74c278e1e4d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:45.835593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1d7494-d588-46c3-b5ae-74c278e1e4d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:45.838340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1d7494-d588-46c3-b5ae-74c278e1e4d8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:45.856628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 403/800] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:45.857901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1d7494-d588-46c3-b5ae-74c278e1e4d8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:45.868375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470a7cba-c3ac-43b7-a07e-fa3b862f0f26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:45.869029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a7cba-c3ac-43b7-a07e-fa3b862f0f26 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:45.869894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a7cba-c3ac-43b7-a07e-fa3b862f0f26 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:45.870175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a7cba-c3ac-43b7-a07e-fa3b862f0f26 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:45.870849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470a7cba-c3ac-43b7-a07e-fa3b862f0f26 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:45.871153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470a7cba-c3ac-43b7-a07e-fa3b862f0f26 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:45.872728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470a7cba-c3ac-43b7-a07e-fa3b862f0f26 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:45.886946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 398/801] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:45.888062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470a7cba-c3ac-43b7-a07e-fa3b862f0f26 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:45.994725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1d7494-d588-46c3-b5ae-74c278e1e4d8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:45.995486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470a7cba-c3ac-43b7-a07e-fa3b862f0f26 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:46.031647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bf7c82-e5e0-44a4-ae36-ea607f08d6ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:46.032645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bf7c82-e5e0-44a4-ae36-ea607f08d6ae None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.033126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bf7c82-e5e0-44a4-ae36-ea607f08d6ae None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.033698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bf7c82-e5e0-44a4-ae36-ea607f08d6ae None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.037541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8669b4ce-875c-475a-b87e-66dc1160fa2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:46.038159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8669b4ce-875c-475a-b87e-66dc1160fa2f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.038438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8669b4ce-875c-475a-b87e-66dc1160fa2f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.038700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8669b4ce-875c-475a-b87e-66dc1160fa2f None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.039445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8669b4ce-875c-475a-b87e-66dc1160fa2f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:46.039852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8669b4ce-875c-475a-b87e-66dc1160fa2f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:46.041327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8669b4ce-875c-475a-b87e-66dc1160fa2f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:46.045107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bf7c82-e5e0-44a4-ae36-ea607f08d6ae None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:46.046819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bf7c82-e5e0-44a4-ae36-ea607f08d6ae None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:46.056209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 404/802] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:46.057235 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8669b4ce-875c-475a-b87e-66dc1160fa2f None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-27bf7c82-e5e0-44a4-ae36-ea607f08d6ae None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:46.062065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:46.063063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:46.063997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:46.063997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:46.063997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:46.063997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:46.063997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:46.063997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:46.063997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:46.063997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:46.063997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:23:46.063997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:46.072434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 399/803] 10.222.0.25 () {56 vars in 1117 bytes} [Mon Sep 26 01:23:45 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:46.073786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bf7c82-e5e0-44a4-ae36-ea607f08d6ae None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:46.084284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:46.088916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:46.189375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8669b4ce-875c-475a-b87e-66dc1160fa2f None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:46.190571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bf7c82-e5e0-44a4-ae36-ea607f08d6ae None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:46.227479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a028019-abc1-4c29-9603-1ad8756c402b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:46.227725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a12228-b5ca-410f-9d7d-66d179bf88fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:46.228105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a028019-abc1-4c29-9603-1ad8756c402b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.228210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a12228-b5ca-410f-9d7d-66d179bf88fa None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.228411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a028019-abc1-4c29-9603-1ad8756c402b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.228523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a12228-b5ca-410f-9d7d-66d179bf88fa None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.228682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a028019-abc1-4c29-9603-1ad8756c402b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.228774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a12228-b5ca-410f-9d7d-66d179bf88fa None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/e542b14838e2466fb59d17e9a20682dd/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.230602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a12228-b5ca-410f-9d7d-66d179bf88fa None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=e542b14838e2466fb59d17e9a20682dd, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:46.232045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a12228-b5ca-410f-9d7d-66d179bf88fa None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:46.240193 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a028019-abc1-4c29-9603-1ad8756c402b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:46.241747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a028019-abc1-4c29-9603-1ad8756c402b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:46.252474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 400/804] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:23:46 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/e542b14838e2466fb59d17e9a20682dd/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:23:46.252474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a12228-b5ca-410f-9d7d-66d179bf88fa None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-3a028019-abc1-4c29-9603-1ad8756c402b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:46.264644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:46.265902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:46.266879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:46.266879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:46.266879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:46.268498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 405/805] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:46.269849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037ba7a-df88-4f22-b278-eb3555f87c77 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.270128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037ba7a-df88-4f22-b278-eb3555f87c77 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.270401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037ba7a-df88-4f22-b278-eb3555f87c77 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.272041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 406/806] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:46.276721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2037ba7a-df88-4f22-b278-eb3555f87c77 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:46.347396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:46.390035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:46.391564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a12228-b5ca-410f-9d7d-66d179bf88fa None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:46.422859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2037ba7a-df88-4f22-b278-eb3555f87c77 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:46.433912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2407680f-7f02-479f-8550-6abadf25d287 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:46.434658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2407680f-7f02-479f-8550-6abadf25d287 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.434933 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2407680f-7f02-479f-8550-6abadf25d287 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.435204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2407680f-7f02-479f-8550-6abadf25d287 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.436004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2407680f-7f02-479f-8550-6abadf25d287 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:46.436394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2407680f-7f02-479f-8550-6abadf25d287 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:46.437861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2407680f-7f02-479f-8550-6abadf25d287 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:46.458427 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 401/807] 10.222.0.25 () {56 vars in 1149 bytes} [Mon Sep 26 01:23:46 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:46.460043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e87a4-0c03-4e65-b48f-364ce49cf40e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.460329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e87a4-0c03-4e65-b48f-364ce49cf40e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.460633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e87a4-0c03-4e65-b48f-364ce49cf40e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.474827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8f3d93-4264-465d-bee7-217b2151b879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:46.475465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8f3d93-4264-465d-bee7-217b2151b879 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.475742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8f3d93-4264-465d-bee7-217b2151b879 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.476065 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8f3d93-4264-465d-bee7-217b2151b879 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.477083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8f3d93-4264-465d-bee7-217b2151b879 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:46.477397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8f3d93-4264-465d-bee7-217b2151b879 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:46.479023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8f3d93-4264-465d-bee7-217b2151b879 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:46.480651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e23e87a4-0c03-4e65-b48f-364ce49cf40e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:46.481845 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e23e87a4-0c03-4e65-b48f-364ce49cf40e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:46.495615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 407/808] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:46.497243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d5d7a5-aa4f-4343-9c2f-300479555603 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.497554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d5d7a5-aa4f-4343-9c2f-300479555603 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.498354 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d5d7a5-aa4f-4343-9c2f-300479555603 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.500068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 408/809] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:46.501064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d5d7a5-aa4f-4343-9c2f-300479555603 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:46.558621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 402/810] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:46 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:46.559896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92c89d7-d092-4949-b6ec-aab62e6ea4ad None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.560260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92c89d7-d092-4949-b6ec-aab62e6ea4ad None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.560636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92c89d7-d092-4949-b6ec-aab62e6ea4ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.583761 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e92c89d7-d092-4949-b6ec-aab62e6ea4ad None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:46.586033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e92c89d7-d092-4949-b6ec-aab62e6ea4ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:46.621514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d5d7a5-aa4f-4343-9c2f-300479555603 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:46.649073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:46.659512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee267a23-27d2-4f3e-99ba-09d88ccca0a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:46.660503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee267a23-27d2-4f3e-99ba-09d88ccca0a2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.660860 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee267a23-27d2-4f3e-99ba-09d88ccca0a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.661232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee267a23-27d2-4f3e-99ba-09d88ccca0a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.663178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:46.672911 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 403/811] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:46 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:46.674152 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92c89d7-d092-4949-b6ec-aab62e6ea4ad None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:46.675711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee267a23-27d2-4f3e-99ba-09d88ccca0a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:46.677432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee267a23-27d2-4f3e-99ba-09d88ccca0a2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ee267a23-27d2-4f3e-99ba-09d88ccca0a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:46.694031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:46.695014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:46.696520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:46.696520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:46.696520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:46.700421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 409/812] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:46.701425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee267a23-27d2-4f3e-99ba-09d88ccca0a2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:46.793910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92c89d7-d092-4949-b6ec-aab62e6ea4ad None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:46.834042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5ffa05-4789-4363-bb0b-b56dff049620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:46.834770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5ffa05-4789-4363-bb0b-b56dff049620 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.835032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5ffa05-4789-4363-bb0b-b56dff049620 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.835314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5ffa05-4789-4363-bb0b-b56dff049620 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.843398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5ffa05-4789-4363-bb0b-b56dff049620 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:46.844905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5ffa05-4789-4363-bb0b-b56dff049620 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4d5ffa05-4789-4363-bb0b-b56dff049620 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:46.859120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:46.860109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:46.861086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 404/813] 10.222.0.25 () {56 vars in 1093 bytes} [Mon Sep 26 01:23:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:46.865855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5ffa05-4789-4363-bb0b-b56dff049620 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:46.882483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee267a23-27d2-4f3e-99ba-09d88ccca0a2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:46.932129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d10a0f5-dfc5-42ec-86c1-5fb3ead65f87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:46.932865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d10a0f5-dfc5-42ec-86c1-5fb3ead65f87 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:46.933256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d10a0f5-dfc5-42ec-86c1-5fb3ead65f87 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:46.933612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d10a0f5-dfc5-42ec-86c1-5fb3ead65f87 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:46.934594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d10a0f5-dfc5-42ec-86c1-5fb3ead65f87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:46.935859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d10a0f5-dfc5-42ec-86c1-5fb3ead65f87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:46.937467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d10a0f5-dfc5-42ec-86c1-5fb3ead65f87 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:46.948764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:46.958092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 410/814] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:46.958092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d10a0f5-dfc5-42ec-86c1-5fb3ead65f87 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:47.014338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:47.015757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5ffa05-4789-4363-bb0b-b56dff049620 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.058336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a4dd78-2e06-4b64-a589-9987ead6d308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:47.059196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a4dd78-2e06-4b64-a589-9987ead6d308 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:47.059647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a4dd78-2e06-4b64-a589-9987ead6d308 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:47.060072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a4dd78-2e06-4b64-a589-9987ead6d308 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:47.060843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a4dd78-2e06-4b64-a589-9987ead6d308 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:47.061173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a4dd78-2e06-4b64-a589-9987ead6d308 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:47.062633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a4dd78-2e06-4b64-a589-9987ead6d308 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:47.076519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d10a0f5-dfc5-42ec-86c1-5fb3ead65f87 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.081446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 405/815] 10.222.0.25 () {56 vars in 1133 bytes} [Mon Sep 26 01:23:46 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:47.087422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a4dd78-2e06-4b64-a589-9987ead6d308 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:47.113274 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99119148-fd0a-4632-bb2e-ecdeb6dceddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:47.113900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99119148-fd0a-4632-bb2e-ecdeb6dceddb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:47.114182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99119148-fd0a-4632-bb2e-ecdeb6dceddb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:47.114626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99119148-fd0a-4632-bb2e-ecdeb6dceddb None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:47.115142 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99119148-fd0a-4632-bb2e-ecdeb6dceddb None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:47.116642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99119148-fd0a-4632-bb2e-ecdeb6dceddb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-99119148-fd0a-4632-bb2e-ecdeb6dceddb None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:47.126712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:47.127734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 26 01:23:47.128611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:47.132510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 411/816] 10.222.0.25 () {56 vars in 1129 bytes} [Mon Sep 26 01:23:46 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:47.134227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99119148-fd0a-4632-bb2e-ecdeb6dceddb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:47.192079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a4dd78-2e06-4b64-a589-9987ead6d308 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.193647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:47.228382 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b572db-800f-4db4-b335-c7008f493540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:47.229027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b572db-800f-4db4-b335-c7008f493540 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:47.229308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b572db-800f-4db4-b335-c7008f493540 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:47.229582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b572db-800f-4db4-b335-c7008f493540 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:47.236889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b572db-800f-4db4-b335-c7008f493540 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:47.238348 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b572db-800f-4db4-b335-c7008f493540 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:47.248024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99119148-fd0a-4632-bb2e-ecdeb6dceddb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e7b572db-800f-4db4-b335-c7008f493540 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:47.251124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:47.252174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:47.253299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:47.253299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:47.253299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:47.253299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:47.253299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:47.253299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:47.253299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:47.253299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:47.257022 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 406/817] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:47.257650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b572db-800f-4db4-b335-c7008f493540 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:47.262082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:47.289015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c9b8e3-7da9-4ce0-b376-85183d3ed01e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:47.290269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c9b8e3-7da9-4ce0-b376-85183d3ed01e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:47.290547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c9b8e3-7da9-4ce0-b376-85183d3ed01e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:47.290817 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c9b8e3-7da9-4ce0-b376-85183d3ed01e None admin] PATH_INFO: `/v3/services/network` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:47.291394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c9b8e3-7da9-4ce0-b376-85183d3ed01e None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:47.292762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c9b8e3-7da9-4ce0-b376-85183d3ed01e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-05c9b8e3-7da9-4ce0-b376-85183d3ed01e None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:47.301385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:47.308044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:47.309233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 412/818] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:47 2022] GET /identity/v3/services/network => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:47.310218 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c9b8e3-7da9-4ce0-b376-85183d3ed01e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:47.373156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b572db-800f-4db4-b335-c7008f493540 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.409226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898c1930-8f18-4abd-86a6-6a6114c60616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:47.410043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898c1930-8f18-4abd-86a6-6a6114c60616 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:47.410326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898c1930-8f18-4abd-86a6-6a6114c60616 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:47.410600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898c1930-8f18-4abd-86a6-6a6114c60616 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:47.421774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c9b8e3-7da9-4ce0-b376-85183d3ed01e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.441406 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:47.458141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c86a1b5-9b5c-41a2-8a79-8364f580b7f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:47.458739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86a1b5-9b5c-41a2-8a79-8364f580b7f2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:47.459023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86a1b5-9b5c-41a2-8a79-8364f580b7f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:47.459299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86a1b5-9b5c-41a2-8a79-8364f580b7f2 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:47.460252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c86a1b5-9b5c-41a2-8a79-8364f580b7f2 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:47.460252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c86a1b5-9b5c-41a2-8a79-8364f580b7f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:47.461665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c86a1b5-9b5c-41a2-8a79-8364f580b7f2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:47.467157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 413/819] 10.222.0.25 () {56 vars in 1124 bytes} [Mon Sep 26 01:23:47 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:47.468267 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c86a1b5-9b5c-41a2-8a79-8364f580b7f2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:47.488392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898c1930-8f18-4abd-86a6-6a6114c60616 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:47.489860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898c1930-8f18-4abd-86a6-6a6114c60616 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:47.512306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:47.577135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c86a1b5-9b5c-41a2-8a79-8364f580b7f2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.599625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 407/820] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:47 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:23:47.599625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898c1930-8f18-4abd-86a6-6a6114c60616 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:47.618761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb82f862-93e4-4023-9b15-f62bca69f8bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:47.619371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb82f862-93e4-4023-9b15-f62bca69f8bb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:47.619647 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb82f862-93e4-4023-9b15-f62bca69f8bb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:47.620195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb82f862-93e4-4023-9b15-f62bca69f8bb None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:47.620628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb82f862-93e4-4023-9b15-f62bca69f8bb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:47.620934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb82f862-93e4-4023-9b15-f62bca69f8bb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:47.622353 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb82f862-93e4-4023-9b15-f62bca69f8bb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:47.635410 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 414/821] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:47.636048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb82f862-93e4-4023-9b15-f62bca69f8bb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:47.667391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:47.707991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898c1930-8f18-4abd-86a6-6a6114c60616 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.737570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:47.742104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb82f862-93e4-4023-9b15-f62bca69f8bb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.743419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c384edf4-264b-41e3-8b62-da8d7ef8cbc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:47.744011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c384edf4-264b-41e3-8b62-da8d7ef8cbc6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:47.744289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c384edf4-264b-41e3-8b62-da8d7ef8cbc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:47.744591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c384edf4-264b-41e3-8b62-da8d7ef8cbc6 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:47.745209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c384edf4-264b-41e3-8b62-da8d7ef8cbc6 None admin] RBAC: Adding query filter params (name=network) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:47.745514 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c384edf4-264b-41e3-8b62-da8d7ef8cbc6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:47.746882 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c384edf4-264b-41e3-8b62-da8d7ef8cbc6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:47.755626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 408/822] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:23:47 2022] GET /identity/v3/services?name=network => generated 124 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:47.755626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c384edf4-264b-41e3-8b62-da8d7ef8cbc6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:47.782342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973a19b0-8d4b-4454-8999-42b278ee842d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:47.783051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973a19b0-8d4b-4454-8999-42b278ee842d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:47.783331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973a19b0-8d4b-4454-8999-42b278ee842d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:47.783603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973a19b0-8d4b-4454-8999-42b278ee842d None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:47.784239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973a19b0-8d4b-4454-8999-42b278ee842d None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:47.784569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973a19b0-8d4b-4454-8999-42b278ee842d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:47.785921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973a19b0-8d4b-4454-8999-42b278ee842d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:47.792233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 415/823] 10.222.0.25 () {56 vars in 1124 bytes} [Mon Sep 26 01:23:47 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:47.793224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973a19b0-8d4b-4454-8999-42b278ee842d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:47.859820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c384edf4-264b-41e3-8b62-da8d7ef8cbc6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.909069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973a19b0-8d4b-4454-8999-42b278ee842d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:47.949081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f48df84-47c5-471a-b7f4-57d32f4d5f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:47.949758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f48df84-47c5-471a-b7f4-57d32f4d5f06 None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:47.950036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f48df84-47c5-471a-b7f4-57d32f4d5f06 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:47.950306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f48df84-47c5-471a-b7f4-57d32f4d5f06 None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/50a8d0782b6f47d1aec027c6e15018a8/roles/03dc4e232ede47a699064dda38338428` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:47.967054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f48df84-47c5-471a-b7f4-57d32f4d5f06 None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=50a8d0782b6f47d1aec027c6e15018a8, role_id=03dc4e232ede47a699064dda38338428)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:47.968542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f48df84-47c5-471a-b7f4-57d32f4d5f06 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:47.996464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 416/824] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:23:47 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/50a8d0782b6f47d1aec027c6e15018a8/roles/03dc4e232ede47a699064dda38338428 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:23:47.997510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f48df84-47c5-471a-b7f4-57d32f4d5f06 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:48.004338 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:48.015828 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c990c920-1ed3-4cc1-827c-0a733cbae09c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:48.016818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c990c920-1ed3-4cc1-827c-0a733cbae09c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:48.017294 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c990c920-1ed3-4cc1-827c-0a733cbae09c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:48.017753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c990c920-1ed3-4cc1-827c-0a733cbae09c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:48.022500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:48.033299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c990c920-1ed3-4cc1-827c-0a733cbae09c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:48.035770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c990c920-1ed3-4cc1-827c-0a733cbae09c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-c990c920-1ed3-4cc1-827c-0a733cbae09c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:48.057936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:48.058974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:48.060433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:48.060433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:48.060433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:48.063499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 409/825] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:48.064772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c990c920-1ed3-4cc1-827c-0a733cbae09c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:48.160162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f48df84-47c5-471a-b7f4-57d32f4d5f06 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:48.192996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c990c920-1ed3-4cc1-827c-0a733cbae09c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:48.212962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e791bdb8-b217-4228-a529-9fe5b4ed7bc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:48.213598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e791bdb8-b217-4228-a529-9fe5b4ed7bc7 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:48.213858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e791bdb8-b217-4228-a529-9fe5b4ed7bc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:48.214131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e791bdb8-b217-4228-a529-9fe5b4ed7bc7 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:48.214766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e791bdb8-b217-4228-a529-9fe5b4ed7bc7 None admin] RBAC: Adding query filter params (type=network) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:48.215072 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e791bdb8-b217-4228-a529-9fe5b4ed7bc7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:48.216472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e791bdb8-b217-4228-a529-9fe5b4ed7bc7 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:48.223003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 417/826] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:23:47 2022] GET /identity/v3/services?type=network => generated 350 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:48.224109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e791bdb8-b217-4228-a529-9fe5b4ed7bc7 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:48.234085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22805779-ec64-41bc-89ae-9e3eaed164de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:48.234930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22805779-ec64-41bc-89ae-9e3eaed164de None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:48.235411 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22805779-ec64-41bc-89ae-9e3eaed164de None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:48.235952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22805779-ec64-41bc-89ae-9e3eaed164de None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:48.236841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22805779-ec64-41bc-89ae-9e3eaed164de None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:48.238192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22805779-ec64-41bc-89ae-9e3eaed164de None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:48.272919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 410/827] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:23:48 2022] POST /identity/v3/endpoints => generated 346 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:48.274582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22805779-ec64-41bc-89ae-9e3eaed164de None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:48.315373 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:48.334247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:48.362547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e791bdb8-b217-4228-a529-9fe5b4ed7bc7 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:48.418729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22805779-ec64-41bc-89ae-9e3eaed164de None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:48.421657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33c3056-0a5f-4052-9bb6-6f342c0f2a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:48.422563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33c3056-0a5f-4052-9bb6-6f342c0f2a61 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:48.422986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33c3056-0a5f-4052-9bb6-6f342c0f2a61 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:48.424458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33c3056-0a5f-4052-9bb6-6f342c0f2a61 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:48.425337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33c3056-0a5f-4052-9bb6-6f342c0f2a61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:48.430680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33c3056-0a5f-4052-9bb6-6f342c0f2a61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:48.432784 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33c3056-0a5f-4052-9bb6-6f342c0f2a61 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:48.459290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 418/828] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:48.467990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33c3056-0a5f-4052-9bb6-6f342c0f2a61 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:48.488250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f419e103-4188-4499-8228-0cf16f7ad289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:48.489304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419e103-4188-4499-8228-0cf16f7ad289 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:48.489823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419e103-4188-4499-8228-0cf16f7ad289 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:48.490348 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419e103-4188-4499-8228-0cf16f7ad289 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:48.493726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f419e103-4188-4499-8228-0cf16f7ad289 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:48.495447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f419e103-4188-4499-8228-0cf16f7ad289 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:48.551633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 411/829] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:23:48 2022] POST /identity/v3/endpoints => generated 337 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:48.556798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931ced81-13bc-4217-9e76-5066b02cf600 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:48.557125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931ced81-13bc-4217-9e76-5066b02cf600 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:48.557363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931ced81-13bc-4217-9e76-5066b02cf600 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:48.559032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 412/830] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:48.564600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a63d865-bd3e-4194-bdd6-d454a380e1de None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:48.564888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a63d865-bd3e-4194-bdd6-d454a380e1de None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:48.565169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a63d865-bd3e-4194-bdd6-d454a380e1de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:48.591962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6a63d865-bd3e-4194-bdd6-d454a380e1de None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:48.593314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6a63d865-bd3e-4194-bdd6-d454a380e1de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:48.619049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33c3056-0a5f-4052-9bb6-6f342c0f2a61 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:48.655212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:48.675565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611781cd-059e-4753-a58b-341e9177eb38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:48.676172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611781cd-059e-4753-a58b-341e9177eb38 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:48.677051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611781cd-059e-4753-a58b-341e9177eb38 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:48.677336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611781cd-059e-4753-a58b-341e9177eb38 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:48.688257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 413/831] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:48 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:23:48.688590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611781cd-059e-4753-a58b-341e9177eb38 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:48.689898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611781cd-059e-4753-a58b-341e9177eb38 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-611781cd-059e-4753-a58b-341e9177eb38 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:48.706910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:48.707887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:48.708825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:48.708825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:48.708825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:48.708825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:48.708825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:48.708825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:48.708825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:48.708825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:48.708825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 419/832] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:48.715031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a63d865-bd3e-4194-bdd6-d454a380e1de None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:48.735757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:48.830284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a63d865-bd3e-4194-bdd6-d454a380e1de None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:48.867903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ebc909-6fa7-43eb-aee8-cad2f09d2a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:48.869297 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ebc909-6fa7-43eb-aee8-cad2f09d2a17 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:48.869750 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ebc909-6fa7-43eb-aee8-cad2f09d2a17 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:48.870189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ebc909-6fa7-43eb-aee8-cad2f09d2a17 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:48.871049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ebc909-6fa7-43eb-aee8-cad2f09d2a17 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:48.871549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ebc909-6fa7-43eb-aee8-cad2f09d2a17 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:48.873266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ebc909-6fa7-43eb-aee8-cad2f09d2a17 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:48.891081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 414/833] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:48.897300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611781cd-059e-4753-a58b-341e9177eb38 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:48.987488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ebc909-6fa7-43eb-aee8-cad2f09d2a17 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:49.032570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611781cd-059e-4753-a58b-341e9177eb38 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:49.080031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a96e46-6e16-494b-a242-ebd55873ebf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:49.080712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a96e46-6e16-494b-a242-ebd55873ebf6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:49.080992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a96e46-6e16-494b-a242-ebd55873ebf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:49.081268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a96e46-6e16-494b-a242-ebd55873ebf6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:49.082286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a96e46-6e16-494b-a242-ebd55873ebf6 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=381ad7c239f444da90369dc58779b86c) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:49.082596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a96e46-6e16-494b-a242-ebd55873ebf6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:49.084199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a96e46-6e16-494b-a242-ebd55873ebf6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:49.092394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 420/834] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:48 2022] GET /identity/v3/role_assignments?user.id=381ad7c239f444da90369dc58779b86c&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:49.104060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:49.132123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ebc909-6fa7-43eb-aee8-cad2f09d2a17 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:49.172148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acdda8b-052f-4aa2-9f8f-8835f7be90c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:49.172843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acdda8b-052f-4aa2-9f8f-8835f7be90c8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:49.173127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acdda8b-052f-4aa2-9f8f-8835f7be90c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:49.173404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acdda8b-052f-4aa2-9f8f-8835f7be90c8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:49.186084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acdda8b-052f-4aa2-9f8f-8835f7be90c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-5acdda8b-052f-4aa2-9f8f-8835f7be90c8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:49.191803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:49.192874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:49.193716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 415/835] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:49.201280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a96e46-6e16-494b-a242-ebd55873ebf6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:49.259295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:49.312621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a96e46-6e16-494b-a242-ebd55873ebf6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:49.348757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af0cb3a-ac7c-45dd-b1df-08e97399d536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:49.349712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af0cb3a-ac7c-45dd-b1df-08e97399d536 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:49.350224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af0cb3a-ac7c-45dd-b1df-08e97399d536 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:49.350700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af0cb3a-ac7c-45dd-b1df-08e97399d536 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:49.352292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af0cb3a-ac7c-45dd-b1df-08e97399d536 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:49.352966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af0cb3a-ac7c-45dd-b1df-08e97399d536 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:49.354534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af0cb3a-ac7c-45dd-b1df-08e97399d536 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:49.365648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 421/836] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:23:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:49.371314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acdda8b-052f-4aa2-9f8f-8835f7be90c8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:49.476251 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acdda8b-052f-4aa2-9f8f-8835f7be90c8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:49.513217 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd08b77-6081-4377-8a3b-1517f6c4f507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:49.513792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd08b77-6081-4377-8a3b-1517f6c4f507 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:49.514064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd08b77-6081-4377-8a3b-1517f6c4f507 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:49.514408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd08b77-6081-4377-8a3b-1517f6c4f507 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:49.517719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:49.527545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd08b77-6081-4377-8a3b-1517f6c4f507 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:49.529044 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd08b77-6081-4377-8a3b-1517f6c4f507 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-cbd08b77-6081-4377-8a3b-1517f6c4f507 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:49.544547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:49.545557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:49.546464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:49.546464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:49.546464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:49.546464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 416/837] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:49.551740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af0cb3a-ac7c-45dd-b1df-08e97399d536 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:49.654497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af0cb3a-ac7c-45dd-b1df-08e97399d536 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:49.669458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:49.691016 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bceccc7-7236-4e7b-a014-fa5afb399e95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:49.691622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bceccc7-7236-4e7b-a014-fa5afb399e95 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:49.691902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bceccc7-7236-4e7b-a014-fa5afb399e95 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:49.692310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bceccc7-7236-4e7b-a014-fa5afb399e95 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:49.692985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bceccc7-7236-4e7b-a014-fa5afb399e95 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:49.693290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bceccc7-7236-4e7b-a014-fa5afb399e95 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:49.694777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bceccc7-7236-4e7b-a014-fa5afb399e95 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:49.707355 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 422/838] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:49.713768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd08b77-6081-4377-8a3b-1517f6c4f507 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:49.818747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd08b77-6081-4377-8a3b-1517f6c4f507 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:49.855142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ec6b3b-2c68-41dd-b418-b0bfb48b6c81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:49.856117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ec6b3b-2c68-41dd-b418-b0bfb48b6c81 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:49.856645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ec6b3b-2c68-41dd-b418-b0bfb48b6c81 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:49.857117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ec6b3b-2c68-41dd-b418-b0bfb48b6c81 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:49.865543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ec6b3b-2c68-41dd-b418-b0bfb48b6c81 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:49.867366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ec6b3b-2c68-41dd-b418-b0bfb48b6c81 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-02ec6b3b-2c68-41dd-b418-b0bfb48b6c81 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:49.879164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:49.880593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:49.881567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:49.881567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:49.881567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:49.881567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:49.881567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:49.881567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:49.881567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:49.881567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:49.881567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 26 01:23:49.881567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:49.885100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 417/839] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:23:49 2022] GET /identity/v3/users/swift => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:49.893443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bceccc7-7236-4e7b-a014-fa5afb399e95 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:50.002713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bceccc7-7236-4e7b-a014-fa5afb399e95 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:50.038381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debbf498-ba8c-43cd-b6b3-a39e213cff22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:50.039063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbf498-ba8c-43cd-b6b3-a39e213cff22 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:50.039620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbf498-ba8c-43cd-b6b3-a39e213cff22 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:50.040071 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debbf498-ba8c-43cd-b6b3-a39e213cff22 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:50.040845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debbf498-ba8c-43cd-b6b3-a39e213cff22 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:50.041153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debbf498-ba8c-43cd-b6b3-a39e213cff22 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:50.042580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debbf498-ba8c-43cd-b6b3-a39e213cff22 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:50.062349 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 423/840] 10.222.0.25 () {56 vars in 1135 bytes} [Mon Sep 26 01:23:49 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:50.067825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ec6b3b-2c68-41dd-b418-b0bfb48b6c81 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:50.120250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:50.180812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ec6b3b-2c68-41dd-b418-b0bfb48b6c81 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:50.215498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e3479f-78e9-4bdd-adf7-af02e9830c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:50.216103 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e3479f-78e9-4bdd-adf7-af02e9830c4a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:50.216403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e3479f-78e9-4bdd-adf7-af02e9830c4a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:50.216680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e3479f-78e9-4bdd-adf7-af02e9830c4a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:50.287345 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e3479f-78e9-4bdd-adf7-af02e9830c4a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:50.288884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e3479f-78e9-4bdd-adf7-af02e9830c4a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:50.342526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:50.395745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 418/841] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:50 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:23:50.399279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b919d633-11b7-4a59-947e-ec58c3173159 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:50.399568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b919d633-11b7-4a59-947e-ec58c3173159 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:50.399849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b919d633-11b7-4a59-947e-ec58c3173159 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:50.405124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 424/842] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:50.405340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e3479f-78e9-4bdd-adf7-af02e9830c4a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:50.406827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb37632-7584-43df-b3a5-1e91f1018871 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:50.407116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb37632-7584-43df-b3a5-1e91f1018871 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:50.407462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb37632-7584-43df-b3a5-1e91f1018871 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:50.426445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-acb37632-7584-43df-b3a5-1e91f1018871 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:50.427700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-acb37632-7584-43df-b3a5-1e91f1018871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:50.456980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:50.502470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 425/843] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:50 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:50.512146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e3479f-78e9-4bdd-adf7-af02e9830c4a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:50.552300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738d1b9c-be8d-47c1-982f-c3949892e17e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:50.553297 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d1b9c-be8d-47c1-982f-c3949892e17e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:50.553874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d1b9c-be8d-47c1-982f-c3949892e17e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:50.554342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d1b9c-be8d-47c1-982f-c3949892e17e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:50.565429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738d1b9c-be8d-47c1-982f-c3949892e17e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:50.567160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738d1b9c-be8d-47c1-982f-c3949892e17e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:50.576944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-738d1b9c-be8d-47c1-982f-c3949892e17e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:50.584122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:50.585074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:50.586540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:50.586540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:50.586540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:50.588610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 419/844] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:50.599171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb37632-7584-43df-b3a5-1e91f1018871 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:50.700477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb37632-7584-43df-b3a5-1e91f1018871 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:50.739092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7237294e-a405-4502-9d95-64bb539f7864 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:50.739730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7237294e-a405-4502-9d95-64bb539f7864 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:50.740004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7237294e-a405-4502-9d95-64bb539f7864 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:50.740277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7237294e-a405-4502-9d95-64bb539f7864 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:50.740970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7237294e-a405-4502-9d95-64bb539f7864 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:50.741272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7237294e-a405-4502-9d95-64bb539f7864 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:50.743622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7237294e-a405-4502-9d95-64bb539f7864 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:50.757772 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 426/845] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:50.765431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738d1b9c-be8d-47c1-982f-c3949892e17e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:50.781998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7237294e-a405-4502-9d95-64bb539f7864 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:50.812989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:50.887650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738d1b9c-be8d-47c1-982f-c3949892e17e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:50.897298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7237294e-a405-4502-9d95-64bb539f7864 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:50.923695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d835bda-28b7-4409-ab89-559ae047e61d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:50.924320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d835bda-28b7-4409-ab89-559ae047e61d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:50.924630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d835bda-28b7-4409-ab89-559ae047e61d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:50.924903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d835bda-28b7-4409-ab89-559ae047e61d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:50.932157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d835bda-28b7-4409-ab89-559ae047e61d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:50.933115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed14920-5371-40f5-b7f2-13dcdecbd9ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:50.933641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d835bda-28b7-4409-ab89-559ae047e61d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:50.933744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed14920-5371-40f5-b7f2-13dcdecbd9ee None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:50.934012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed14920-5371-40f5-b7f2-13dcdecbd9ee None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:50.934278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed14920-5371-40f5-b7f2-13dcdecbd9ee None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-7d835bda-28b7-4409-ab89-559ae047e61d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:50.944768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:50.945822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:50.946749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:50.946749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:50.946749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:50.946749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:50.946749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:50.946749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:50.946749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:50.946749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:50.947173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed14920-5371-40f5-b7f2-13dcdecbd9ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:50.950896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 420/846] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-0ed14920-5371-40f5-b7f2-13dcdecbd9ee None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:50.951628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:50.952758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:23:50.953604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:50.954256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df93a57-f8d8-4671-a69a-e9e2ae81d12f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:50.954256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df93a57-f8d8-4671-a69a-e9e2ae81d12f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:50.954256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df93a57-f8d8-4671-a69a-e9e2ae81d12f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:50.957773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 427/847] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:23:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:50.958325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 421/848] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:50 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:50.958899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed14920-5371-40f5-b7f2-13dcdecbd9ee None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:50.959407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4664c85e-f8ac-491e-a9dc-48b7d963ff7d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:50.959698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4664c85e-f8ac-491e-a9dc-48b7d963ff7d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:50.959957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4664c85e-f8ac-491e-a9dc-48b7d963ff7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:50.978431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4664c85e-f8ac-491e-a9dc-48b7d963ff7d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:50.979614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4664c85e-f8ac-491e-a9dc-48b7d963ff7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:51.042330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:51.051705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:51.064664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 422/849] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:50 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:51.066341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4664c85e-f8ac-491e-a9dc-48b7d963ff7d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:51.073294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed14920-5371-40f5-b7f2-13dcdecbd9ee None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:51.116138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f7fb11-c777-47cf-a863-9a19b31a63a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:51.116832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f7fb11-c777-47cf-a863-9a19b31a63a4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.117179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f7fb11-c777-47cf-a863-9a19b31a63a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.117571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f7fb11-c777-47cf-a863-9a19b31a63a4 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.118277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f7fb11-c777-47cf-a863-9a19b31a63a4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:51.118585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f7fb11-c777-47cf-a863-9a19b31a63a4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:51.120527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f7fb11-c777-47cf-a863-9a19b31a63a4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:51.137271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 428/850] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:51.143313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f7fb11-c777-47cf-a863-9a19b31a63a4 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:51.193596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4664c85e-f8ac-491e-a9dc-48b7d963ff7d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:51.231616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447c1d47-94e0-4bf4-a663-5ef427ede1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:51.232380 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447c1d47-94e0-4bf4-a663-5ef427ede1e3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.232778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447c1d47-94e0-4bf4-a663-5ef427ede1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.233152 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447c1d47-94e0-4bf4-a663-5ef427ede1e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.233918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447c1d47-94e0-4bf4-a663-5ef427ede1e3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:51.234325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447c1d47-94e0-4bf4-a663-5ef427ede1e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:51.235795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447c1d47-94e0-4bf4-a663-5ef427ede1e3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:51.248385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 423/851] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:23:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:51.252502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447c1d47-94e0-4bf4-a663-5ef427ede1e3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:51.273058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f7fb11-c777-47cf-a863-9a19b31a63a4 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:51.312186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:51.320052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42300ff9-1859-4792-af94-f5c139a697e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:51.322209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42300ff9-1859-4792-af94-f5c139a697e4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.323012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42300ff9-1859-4792-af94-f5c139a697e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.323586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42300ff9-1859-4792-af94-f5c139a697e4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.324552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42300ff9-1859-4792-af94-f5c139a697e4 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=e542b14838e2466fb59d17e9a20682dd) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:51.325072 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42300ff9-1859-4792-af94-f5c139a697e4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:51.326992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42300ff9-1859-4792-af94-f5c139a697e4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:51.335056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 429/852] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:51 2022] GET /identity/v3/role_assignments?user.id=e542b14838e2466fb59d17e9a20682dd&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:51.336963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42300ff9-1859-4792-af94-f5c139a697e4 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:51.347502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:51.371974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447c1d47-94e0-4bf4-a663-5ef427ede1e3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:51.415098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4248b1bc-b64f-4f9d-89cf-5a0562349a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:51.416190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4248b1bc-b64f-4f9d-89cf-5a0562349a29 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.416693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4248b1bc-b64f-4f9d-89cf-5a0562349a29 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.417175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4248b1bc-b64f-4f9d-89cf-5a0562349a29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.431297 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4248b1bc-b64f-4f9d-89cf-5a0562349a29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:51.433237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4248b1bc-b64f-4f9d-89cf-5a0562349a29 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4248b1bc-b64f-4f9d-89cf-5a0562349a29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:51.451247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:51.452269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:51.453540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:51.453540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:51.453540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:51.460478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 424/853] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:51.462751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4248b1bc-b64f-4f9d-89cf-5a0562349a29 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:51.468285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42300ff9-1859-4792-af94-f5c139a697e4 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:51.510407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13976f-5fb6-42b2-9ee6-a11cedbc7bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:51.510968 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13976f-5fb6-42b2-9ee6-a11cedbc7bb4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.511248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13976f-5fb6-42b2-9ee6-a11cedbc7bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.511523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13976f-5fb6-42b2-9ee6-a11cedbc7bb4 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.512375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b13976f-5fb6-42b2-9ee6-a11cedbc7bb4 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:51.513648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b13976f-5fb6-42b2-9ee6-a11cedbc7bb4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-0b13976f-5fb6-42b2-9ee6-a11cedbc7bb4 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:51.525528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:51.527341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 26 01:23:51.528459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:51.529456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 430/854] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:23:51 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:51.533594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13976f-5fb6-42b2-9ee6-a11cedbc7bb4 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:51.587434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4248b1bc-b64f-4f9d-89cf-5a0562349a29 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:51.589868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:51.611867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:51.625821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ae6d3d-49db-4f79-a8bb-e2b668ab3882 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:51.626421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae6d3d-49db-4f79-a8bb-e2b668ab3882 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.626697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae6d3d-49db-4f79-a8bb-e2b668ab3882 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.626968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae6d3d-49db-4f79-a8bb-e2b668ab3882 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.627628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ae6d3d-49db-4f79-a8bb-e2b668ab3882 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:51.627897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ae6d3d-49db-4f79-a8bb-e2b668ab3882 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:51.629288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ae6d3d-49db-4f79-a8bb-e2b668ab3882 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:51.647009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 425/855] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:51.647009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdb66e9-d719-4d5d-b89b-e6bf1105bcea None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.647009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdb66e9-d719-4d5d-b89b-e6bf1105bcea None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.647009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdb66e9-d719-4d5d-b89b-e6bf1105bcea None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.647009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 426/856] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:51 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:51.650239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdb66e9-d719-4d5d-b89b-e6bf1105bcea None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:51.654637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13976f-5fb6-42b2-9ee6-a11cedbc7bb4 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:51.693668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e680f47d-2881-4927-bec4-fa49067b1d55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:51.694432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e680f47d-2881-4927-bec4-fa49067b1d55 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.694715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e680f47d-2881-4927-bec4-fa49067b1d55 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.694991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e680f47d-2881-4927-bec4-fa49067b1d55 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.696531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e680f47d-2881-4927-bec4-fa49067b1d55 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:51.696953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e680f47d-2881-4927-bec4-fa49067b1d55 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:51.698337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e680f47d-2881-4927-bec4-fa49067b1d55 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:51.707965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 431/857] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:23:51 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:51.707965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59550866-7d9e-4845-a83e-edcfa0ad22b0 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.707965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59550866-7d9e-4845-a83e-edcfa0ad22b0 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.707965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59550866-7d9e-4845-a83e-edcfa0ad22b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.725135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-59550866-7d9e-4845-a83e-edcfa0ad22b0 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:51.726489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-59550866-7d9e-4845-a83e-edcfa0ad22b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:51.757198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdb66e9-d719-4d5d-b89b-e6bf1105bcea None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:51.795946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3095dfa-e58f-4280-bee3-260fbdbc30d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:51.796527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3095dfa-e58f-4280-bee3-260fbdbc30d0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.796816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3095dfa-e58f-4280-bee3-260fbdbc30d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.797098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3095dfa-e58f-4280-bee3-260fbdbc30d0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.803265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 432/858] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:51 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:51.804283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59550866-7d9e-4845-a83e-edcfa0ad22b0 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:51.814292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3095dfa-e58f-4280-bee3-260fbdbc30d0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:51.816206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3095dfa-e58f-4280-bee3-260fbdbc30d0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e3095dfa-e58f-4280-bee3-260fbdbc30d0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:51.832703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:51.833850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:51.835020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 427/859] 10.222.0.25 () {56 vars in 1117 bytes} [Mon Sep 26 01:23:51 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:51.841120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3095dfa-e58f-4280-bee3-260fbdbc30d0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:51.879292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:51.883848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:51.933023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59550866-7d9e-4845-a83e-edcfa0ad22b0 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:51.963657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3095dfa-e58f-4280-bee3-260fbdbc30d0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:51.979323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754e2f13-371d-43b0-866d-dd24992703b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:51.979942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e2f13-371d-43b0-866d-dd24992703b4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:51.980225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e2f13-371d-43b0-866d-dd24992703b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:51.980800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e2f13-371d-43b0-866d-dd24992703b4 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:51.981458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754e2f13-371d-43b0-866d-dd24992703b4 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:51.981766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754e2f13-371d-43b0-866d-dd24992703b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:51.983773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754e2f13-371d-43b0-866d-dd24992703b4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:51.989796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 433/860] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:23:51 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:52.006168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a03ffd6-d8b5-40fd-8189-558ff1d8d55f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:52.007248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03ffd6-d8b5-40fd-8189-558ff1d8d55f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:52.008330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03ffd6-d8b5-40fd-8189-558ff1d8d55f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:52.009037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03ffd6-d8b5-40fd-8189-558ff1d8d55f None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:52.010151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a03ffd6-d8b5-40fd-8189-558ff1d8d55f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:52.010675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a03ffd6-d8b5-40fd-8189-558ff1d8d55f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:52.012563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a03ffd6-d8b5-40fd-8189-558ff1d8d55f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:52.035617 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 428/861] 10.222.0.25 () {56 vars in 1149 bytes} [Mon Sep 26 01:23:51 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:52.041051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754e2f13-371d-43b0-866d-dd24992703b4 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:52.116456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a03ffd6-d8b5-40fd-8189-558ff1d8d55f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:52.180507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754e2f13-371d-43b0-866d-dd24992703b4 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:52.211701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:52.220609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e4578e-0599-4c29-8d05-0c53edf0004c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:52.221225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e4578e-0599-4c29-8d05-0c53edf0004c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:52.221504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e4578e-0599-4c29-8d05-0c53edf0004c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:52.221775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e4578e-0599-4c29-8d05-0c53edf0004c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:52.224490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:52.248216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a03ffd6-d8b5-40fd-8189-558ff1d8d55f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:52.301866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5945cd26-d323-4e7f-b375-a7434a46e5a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:52.302435 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5945cd26-d323-4e7f-b375-a7434a46e5a3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:52.302725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5945cd26-d323-4e7f-b375-a7434a46e5a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:52.303227 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5945cd26-d323-4e7f-b375-a7434a46e5a3 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:52.303825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5945cd26-d323-4e7f-b375-a7434a46e5a3 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:52.305876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5945cd26-d323-4e7f-b375-a7434a46e5a3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:52.307547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e4578e-0599-4c29-8d05-0c53edf0004c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:52.310346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e4578e-0599-4c29-8d05-0c53edf0004c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-5945cd26-d323-4e7f-b375-a7434a46e5a3 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:52.316450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:52.317482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 26 01:23:52.318438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:52.321374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 429/862] 10.222.0.25 () {56 vars in 1132 bytes} [Mon Sep 26 01:23:52 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:52.327433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5945cd26-d323-4e7f-b375-a7434a46e5a3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:52.436420 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 434/863] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:52 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:23:52.439885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e4578e-0599-4c29-8d05-0c53edf0004c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:52.460669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5945cd26-d323-4e7f-b375-a7434a46e5a3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:52.469758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:52.497355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b13f1d-9b30-43cf-ab9a-5ececde49478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:52.498326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b13f1d-9b30-43cf-ab9a-5ececde49478 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:52.498824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b13f1d-9b30-43cf-ab9a-5ececde49478 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:52.499336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b13f1d-9b30-43cf-ab9a-5ececde49478 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:52.500167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b13f1d-9b30-43cf-ab9a-5ececde49478 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:52.500695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b13f1d-9b30-43cf-ab9a-5ececde49478 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:52.502323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b13f1d-9b30-43cf-ab9a-5ececde49478 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:52.507614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 430/864] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:23:52 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:52.514016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b13f1d-9b30-43cf-ab9a-5ececde49478 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:52.536833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:52.549819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e4578e-0599-4c29-8d05-0c53edf0004c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:52.588089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b8998a-009b-4bda-bfb1-1533941e58c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:52.588834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b8998a-009b-4bda-bfb1-1533941e58c6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:52.589546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b8998a-009b-4bda-bfb1-1533941e58c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:52.589546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b8998a-009b-4bda-bfb1-1533941e58c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:52.600669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b8998a-009b-4bda-bfb1-1533941e58c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:52.602924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b8998a-009b-4bda-bfb1-1533941e58c6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-42b8998a-009b-4bda-bfb1-1533941e58c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:52.619695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:52.620767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:52.622489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:52.622489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:52.622489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:52.622489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 435/865] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:52.627356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b8998a-009b-4bda-bfb1-1533941e58c6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:52.660052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b13f1d-9b30-43cf-ab9a-5ececde49478 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:52.699063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbceb6bf-3046-4306-8a10-e3cc1780839f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:52.699764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbceb6bf-3046-4306-8a10-e3cc1780839f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:52.700107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbceb6bf-3046-4306-8a10-e3cc1780839f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:52.700475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbceb6bf-3046-4306-8a10-e3cc1780839f None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:52.701186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbceb6bf-3046-4306-8a10-e3cc1780839f None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:52.701553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbceb6bf-3046-4306-8a10-e3cc1780839f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:52.703040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbceb6bf-3046-4306-8a10-e3cc1780839f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:52.708360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 431/866] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:23:52 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:52.729409 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:52.752688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b8998a-009b-4bda-bfb1-1533941e58c6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:52.793270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b454db55-98de-433e-88f1-3786c53b6b31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:52.793891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454db55-98de-433e-88f1-3786c53b6b31 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:52.794235 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454db55-98de-433e-88f1-3786c53b6b31 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:52.794513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454db55-98de-433e-88f1-3786c53b6b31 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:52.795170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b454db55-98de-433e-88f1-3786c53b6b31 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:52.795468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b454db55-98de-433e-88f1-3786c53b6b31 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:52.796907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b454db55-98de-433e-88f1-3786c53b6b31 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:52.812154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 436/867] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:52.817675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbceb6bf-3046-4306-8a10-e3cc1780839f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:52.903646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:52.931640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbceb6bf-3046-4306-8a10-e3cc1780839f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:52.970132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e61619-cb85-4a04-b55f-84c90d8626a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:52.970785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e61619-cb85-4a04-b55f-84c90d8626a9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:52.971029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e61619-cb85-4a04-b55f-84c90d8626a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:52.971301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e61619-cb85-4a04-b55f-84c90d8626a9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:52.978724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e61619-cb85-4a04-b55f-84c90d8626a9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:52.980163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e61619-cb85-4a04-b55f-84c90d8626a9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-10e61619-cb85-4a04-b55f-84c90d8626a9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:52.991838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:52.993640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:52.994923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:52.994923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:52.994923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:52.994923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:52.994923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:52.994923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:52.994923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:52.994923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:52.994923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 432/868] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:53.001430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b454db55-98de-433e-88f1-3786c53b6b31 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:53.111574 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b454db55-98de-433e-88f1-3786c53b6b31 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:53.146733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fee117-2142-45f7-9878-867320e328f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:53.147329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fee117-2142-45f7-9878-867320e328f0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:53.147606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fee117-2142-45f7-9878-867320e328f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:53.147876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fee117-2142-45f7-9878-867320e328f0 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:53.149302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fee117-2142-45f7-9878-867320e328f0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:53.149779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fee117-2142-45f7-9878-867320e328f0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:53.151201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fee117-2142-45f7-9878-867320e328f0 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:53.159854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:53.163621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 437/869] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:53.170162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e61619-cb85-4a04-b55f-84c90d8626a9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:53.277391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e61619-cb85-4a04-b55f-84c90d8626a9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:53.314039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444129eb-a172-47d1-8c51-40ef35b350ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:53.314645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444129eb-a172-47d1-8c51-40ef35b350ab None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:53.314925 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444129eb-a172-47d1-8c51-40ef35b350ab None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:53.315202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444129eb-a172-47d1-8c51-40ef35b350ab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:53.315931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444129eb-a172-47d1-8c51-40ef35b350ab None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=50a8d0782b6f47d1aec027c6e15018a8) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:53.316240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444129eb-a172-47d1-8c51-40ef35b350ab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:53.317708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444129eb-a172-47d1-8c51-40ef35b350ab None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:53.319447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:53.326732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 433/870] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:53 2022] GET /identity/v3/role_assignments?user.id=50a8d0782b6f47d1aec027c6e15018a8&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:53.627180 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4005cd-c31a-4572-8e33-83088b1380f0 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:53.627724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4005cd-c31a-4572-8e33-83088b1380f0 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:53.627724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4005cd-c31a-4572-8e33-83088b1380f0 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:53.629360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 438/871] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:53.635468 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2ef201-1087-40fc-82a0-632b4e94a31f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:53.635747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2ef201-1087-40fc-82a0-632b4e94a31f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:53.636017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2ef201-1087-40fc-82a0-632b4e94a31f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:53.654532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6e2ef201-1087-40fc-82a0-632b4e94a31f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:53.655718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6e2ef201-1087-40fc-82a0-632b4e94a31f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:53.733206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 434/872] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:53 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:54.011128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4005cd-c31a-4572-8e33-83088b1380f0 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:54.116650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4005cd-c31a-4572-8e33-83088b1380f0 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:54.156069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5701b6ec-6012-49c3-a8c3-5c74c1b7994c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:54.156725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5701b6ec-6012-49c3-a8c3-5c74c1b7994c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:54.156997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5701b6ec-6012-49c3-a8c3-5c74c1b7994c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:54.157263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5701b6ec-6012-49c3-a8c3-5c74c1b7994c None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:54.157852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5701b6ec-6012-49c3-a8c3-5c74c1b7994c None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:54.159295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5701b6ec-6012-49c3-a8c3-5c74c1b7994c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-5701b6ec-6012-49c3-a8c3-5c74c1b7994c None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:54.169845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:54.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:54.172115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 439/873] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:23:54 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:54.177342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2ef201-1087-40fc-82a0-632b4e94a31f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:54.285257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2ef201-1087-40fc-82a0-632b4e94a31f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:54.315563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:54.328375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbea261b-a4d9-4328-b0e4-1db050ad143f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:54.328375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbea261b-a4d9-4328-b0e4-1db050ad143f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:54.328375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbea261b-a4d9-4328-b0e4-1db050ad143f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:54.328740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546555a9-ec6e-4324-be82-76339c7e0792 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:54.328830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbea261b-a4d9-4328-b0e4-1db050ad143f None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:54.328911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546555a9-ec6e-4324-be82-76339c7e0792 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:54.328911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546555a9-ec6e-4324-be82-76339c7e0792 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:54.329045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbea261b-a4d9-4328-b0e4-1db050ad143f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:54.329045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbea261b-a4d9-4328-b0e4-1db050ad143f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:54.329408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbea261b-a4d9-4328-b0e4-1db050ad143f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:54.331370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 440/874] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:23:54.335893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 435/875] 10.222.0.25 () {56 vars in 1122 bytes} [Mon Sep 26 01:23:54 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:54.341910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2285caa4-b29c-48a1-bd60-8f31e057057e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:54.341910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2285caa4-b29c-48a1-bd60-8f31e057057e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:54.341910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2285caa4-b29c-48a1-bd60-8f31e057057e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:54.342191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbea261b-a4d9-4328-b0e4-1db050ad143f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:54.361823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-2285caa4-b29c-48a1-bd60-8f31e057057e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:54.363557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2285caa4-b29c-48a1-bd60-8f31e057057e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:54.444737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:54.455165 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbea261b-a4d9-4328-b0e4-1db050ad143f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:54.460417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 441/876] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:54 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:23:54.501685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94553148-8896-4832-a489-13116517fbbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:54.502287 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94553148-8896-4832-a489-13116517fbbf None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:54.502570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94553148-8896-4832-a489-13116517fbbf None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:54.502846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94553148-8896-4832-a489-13116517fbbf None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:54.503477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94553148-8896-4832-a489-13116517fbbf None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:54.503779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94553148-8896-4832-a489-13116517fbbf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:54.505174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94553148-8896-4832-a489-13116517fbbf None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:54.510811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 436/877] 10.222.0.25 () {56 vars in 1122 bytes} [Mon Sep 26 01:23:54 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:54.744975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2285caa4-b29c-48a1-bd60-8f31e057057e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:54.850942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2285caa4-b29c-48a1-bd60-8f31e057057e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:54.887573 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a797f6-e427-4f1d-9ce2-6a8716a9a3d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:54.888663 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a797f6-e427-4f1d-9ce2-6a8716a9a3d4 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:54.889166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a797f6-e427-4f1d-9ce2-6a8716a9a3d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:54.889707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a797f6-e427-4f1d-9ce2-6a8716a9a3d4 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:54.890624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a797f6-e427-4f1d-9ce2-6a8716a9a3d4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:54.892266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a797f6-e427-4f1d-9ce2-6a8716a9a3d4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:54.909423 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 442/878] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:23:54 2022] POST /identity/v3/services => generated 251 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:55.019399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e01502-666a-459a-a80b-1103133bbf0a None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:55.019574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e01502-666a-459a-a80b-1103133bbf0a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:55.019847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e01502-666a-459a-a80b-1103133bbf0a None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:55.021487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 437/879] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:55.027628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358f9f6b-721d-4f56-b01f-4ec5a48bdec6 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:55.027941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358f9f6b-721d-4f56-b01f-4ec5a48bdec6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:55.028183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358f9f6b-721d-4f56-b01f-4ec5a48bdec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:55.052045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-358f9f6b-721d-4f56-b01f-4ec5a48bdec6 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:55.053371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-358f9f6b-721d-4f56-b01f-4ec5a48bdec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:55.113163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:55.139277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 443/880] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:55 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:55.408321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e01502-666a-459a-a80b-1103133bbf0a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:55.472887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:55.520529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e01502-666a-459a-a80b-1103133bbf0a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:55.556181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd0d84f-207e-4295-a543-baacd5ad06be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:55.556841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0d84f-207e-4295-a543-baacd5ad06be None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:55.557118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0d84f-207e-4295-a543-baacd5ad06be None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:55.557391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0d84f-207e-4295-a543-baacd5ad06be None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:55.558081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd0d84f-207e-4295-a543-baacd5ad06be None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:55.559633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd0d84f-207e-4295-a543-baacd5ad06be None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:55.579516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 438/881] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:23:55 2022] POST /identity/v3/services => generated 269 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:23:55.691007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14941327-069e-46ae-955c-38fcab17c30c None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:55.691310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14941327-069e-46ae-955c-38fcab17c30c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:55.691516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14941327-069e-46ae-955c-38fcab17c30c None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:55.693128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 444/882] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:55.699366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77eb0ff-6e38-4028-a173-7d24418e13ec None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:55.699942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77eb0ff-6e38-4028-a173-7d24418e13ec None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:55.700489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77eb0ff-6e38-4028-a173-7d24418e13ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:55.720792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d77eb0ff-6e38-4028-a173-7d24418e13ec None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:55.722426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d77eb0ff-6e38-4028-a173-7d24418e13ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:55.802533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 439/883] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:55 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:56.076951 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14941327-069e-46ae-955c-38fcab17c30c None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:56.193925 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14941327-069e-46ae-955c-38fcab17c30c None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:56.230054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef69e94d-c36c-45b7-9ca4-319e7f457316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:56.231035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef69e94d-c36c-45b7-9ca4-319e7f457316 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:56.231510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef69e94d-c36c-45b7-9ca4-319e7f457316 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:56.231978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef69e94d-c36c-45b7-9ca4-319e7f457316 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:56.243712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef69e94d-c36c-45b7-9ca4-319e7f457316 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-ef69e94d-c36c-45b7-9ca4-319e7f457316 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:23:56.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:56.249761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:23:56.250604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:56.253292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 445/884] 10.222.0.25 () {56 vars in 1120 bytes} [Mon Sep 26 01:23:56 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:56.259458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77eb0ff-6e38-4028-a173-7d24418e13ec None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:56.304243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:56.364361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77eb0ff-6e38-4028-a173-7d24418e13ec None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:56.400120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23d9d27-8d2f-40d8-a50d-3d69bff92645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:56.400773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23d9d27-8d2f-40d8-a50d-3d69bff92645 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:56.401054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23d9d27-8d2f-40d8-a50d-3d69bff92645 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:56.401328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23d9d27-8d2f-40d8-a50d-3d69bff92645 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:56.401992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23d9d27-8d2f-40d8-a50d-3d69bff92645 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:56.402295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23d9d27-8d2f-40d8-a50d-3d69bff92645 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:56.403667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23d9d27-8d2f-40d8-a50d-3d69bff92645 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:56.412016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 440/885] 10.222.0.25 () {56 vars in 1115 bytes} [Mon Sep 26 01:23:56 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:56.418563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef69e94d-c36c-45b7-9ca4-319e7f457316 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:56.463819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:56.523895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef69e94d-c36c-45b7-9ca4-319e7f457316 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:56.561998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca5aa77-b868-4fd3-a3d0-df2046b89d5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:56.562566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca5aa77-b868-4fd3-a3d0-df2046b89d5e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:56.562841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca5aa77-b868-4fd3-a3d0-df2046b89d5e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:56.563112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca5aa77-b868-4fd3-a3d0-df2046b89d5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:56.574960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca5aa77-b868-4fd3-a3d0-df2046b89d5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:56.576440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca5aa77-b868-4fd3-a3d0-df2046b89d5e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-8ca5aa77-b868-4fd3-a3d0-df2046b89d5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:56.592467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:56.593504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:56.594478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:56.594478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:56.594478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:56.594478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 446/886] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:56.599865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23d9d27-8d2f-40d8-a50d-3d69bff92645 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:56.709836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23d9d27-8d2f-40d8-a50d-3d69bff92645 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:56.713328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:56.726039 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59974e-7f2c-4139-995c-09abe0ee5060 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:56.726432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59974e-7f2c-4139-995c-09abe0ee5060 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:56.726907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59974e-7f2c-4139-995c-09abe0ee5060 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:56.728738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 447/887] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:56.733896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0709b636-fd7a-4fee-86f1-4da7b1298195 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:56.734179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0709b636-fd7a-4fee-86f1-4da7b1298195 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:56.734455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0709b636-fd7a-4fee-86f1-4da7b1298195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:56.746598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693ed61f-10ff-489e-ba42-435b659f839d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:56.747257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693ed61f-10ff-489e-ba42-435b659f839d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:56.747546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693ed61f-10ff-489e-ba42-435b659f839d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:56.747821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693ed61f-10ff-489e-ba42-435b659f839d None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:56.748613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693ed61f-10ff-489e-ba42-435b659f839d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:56.748924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693ed61f-10ff-489e-ba42-435b659f839d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:56.750414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693ed61f-10ff-489e-ba42-435b659f839d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:56.753660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0709b636-fd7a-4fee-86f1-4da7b1298195 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:56.754843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0709b636-fd7a-4fee-86f1-4da7b1298195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:56.763429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 441/888] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:56.777330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693ed61f-10ff-489e-ba42-435b659f839d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:56.826995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 448/889] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:56 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:23:56.884234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693ed61f-10ff-489e-ba42-435b659f839d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:56.918711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ca815d-73ff-493e-858e-fd80d6f7ad62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:56.919758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ca815d-73ff-493e-858e-fd80d6f7ad62 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:56.920234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ca815d-73ff-493e-858e-fd80d6f7ad62 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:56.920781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ca815d-73ff-493e-858e-fd80d6f7ad62 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:56.928401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ca815d-73ff-493e-858e-fd80d6f7ad62 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:56.930685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ca815d-73ff-493e-858e-fd80d6f7ad62 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d5ca815d-73ff-493e-858e-fd80d6f7ad62 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:23:56.943120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:56.944118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:56.945222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:56.945222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:56.945222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:23:56.945222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:23:56.945222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:23:56.945222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:23:56.945222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:23:56.945222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:23:56.945222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:23:56.945222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:56.947869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 442/890] 10.222.0.25 () {56 vars in 1117 bytes} [Mon Sep 26 01:23:56 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:56.953914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0709b636-fd7a-4fee-86f1-4da7b1298195 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:56.980816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:57.058698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0709b636-fd7a-4fee-86f1-4da7b1298195 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:57.096006 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6220895b-98a2-4d0c-9d73-f15ac4af9c32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:57.096640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6220895b-98a2-4d0c-9d73-f15ac4af9c32 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:57.096855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6220895b-98a2-4d0c-9d73-f15ac4af9c32 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:57.097107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6220895b-98a2-4d0c-9d73-f15ac4af9c32 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:57.097845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6220895b-98a2-4d0c-9d73-f15ac4af9c32 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:57.098146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6220895b-98a2-4d0c-9d73-f15ac4af9c32 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:57.099600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6220895b-98a2-4d0c-9d73-f15ac4af9c32 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:57.121040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 449/891] 10.222.0.25 () {56 vars in 1149 bytes} [Mon Sep 26 01:23:56 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:57.121309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ca815d-73ff-493e-858e-fd80d6f7ad62 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:57.129469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6220895b-98a2-4d0c-9d73-f15ac4af9c32 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:57.160795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:57.224166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:57.245233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ca815d-73ff-493e-858e-fd80d6f7ad62 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:57.245422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6220895b-98a2-4d0c-9d73-f15ac4af9c32 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:57.279989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6bc805-250a-4d0c-8dde-403d345c5e1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:57.280730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6bc805-250a-4d0c-8dde-403d345c5e1c None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:57.281008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6bc805-250a-4d0c-8dde-403d345c5e1c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:57.281283 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6bc805-250a-4d0c-8dde-403d345c5e1c None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:57.281826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d6558e-5df8-45c9-a483-dffbf921614b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:57.281944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6bc805-250a-4d0c-8dde-403d345c5e1c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:57.282427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d6558e-5df8-45c9-a483-dffbf921614b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:57.282729 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d6558e-5df8-45c9-a483-dffbf921614b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:57.283107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d6558e-5df8-45c9-a483-dffbf921614b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:57.283286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6bc805-250a-4d0c-8dde-403d345c5e1c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:57.300414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 443/892] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:23:57 2022] POST /identity/v3/services => generated 241 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:23:57.300939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe4a68f-2fa9-42f5-9f1c-45610085ff5a None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:57.301222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe4a68f-2fa9-42f5-9f1c-45610085ff5a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:57.301496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe4a68f-2fa9-42f5-9f1c-45610085ff5a None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:57.302877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 444/893] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:57.309084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace2ca14-aed6-4ca8-8bab-1653f0c952e3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:57.309375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace2ca14-aed6-4ca8-8bab-1653f0c952e3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:57.309655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace2ca14-aed6-4ca8-8bab-1653f0c952e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:57.331115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ace2ca14-aed6-4ca8-8bab-1653f0c952e3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:57.333144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ace2ca14-aed6-4ca8-8bab-1653f0c952e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:57.376631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d6558e-5df8-45c9-a483-dffbf921614b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:57.378034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d6558e-5df8-45c9-a483-dffbf921614b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:57.451146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 445/894] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:57 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:23:57.477701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:57.499929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 450/895] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:23:57 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:23:57.507360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace2ca14-aed6-4ca8-8bab-1653f0c952e3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:57.520944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:57.614439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace2ca14-aed6-4ca8-8bab-1653f0c952e3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:57.650132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf19ed8a-4cca-437d-b7a8-cdbd3dc745ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:57.650735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf19ed8a-4cca-437d-b7a8-cdbd3dc745ec None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:57.651012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf19ed8a-4cca-437d-b7a8-cdbd3dc745ec None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:57.651281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf19ed8a-4cca-437d-b7a8-cdbd3dc745ec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:57.661739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf19ed8a-4cca-437d-b7a8-cdbd3dc745ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:57.663252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf19ed8a-4cca-437d-b7a8-cdbd3dc745ec None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-cf19ed8a-4cca-437d-b7a8-cdbd3dc745ec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:23:57.678672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:57.680491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:23:57.681608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:23:57.681608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:23:57.681608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:57.681608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 446/896] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:23:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:57.690146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d6558e-5df8-45c9-a483-dffbf921614b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:57.748283 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf19ed8a-4cca-437d-b7a8-cdbd3dc745ec None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:57.805375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d6558e-5df8-45c9-a483-dffbf921614b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:57.824732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:57.847662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d59cc0-2b26-4c0f-80ba-ab66d4a6afc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:57.848754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d59cc0-2b26-4c0f-80ba-ab66d4a6afc5 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:57.849262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d59cc0-2b26-4c0f-80ba-ab66d4a6afc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:57.849740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d59cc0-2b26-4c0f-80ba-ab66d4a6afc5 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:57.850616 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d59cc0-2b26-4c0f-80ba-ab66d4a6afc5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:57.851940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d59cc0-2b26-4c0f-80ba-ab66d4a6afc5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:57.854451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d59cc0-2b26-4c0f-80ba-ab66d4a6afc5 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:57.863278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf19ed8a-4cca-437d-b7a8-cdbd3dc745ec None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:57.872438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 451/897] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:23:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:57.877676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d59cc0-2b26-4c0f-80ba-ab66d4a6afc5 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:57.905209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fe6ba2-78a7-417e-a5be-4f3574d97fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:57.906182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fe6ba2-78a7-417e-a5be-4f3574d97fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:57.906652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fe6ba2-78a7-417e-a5be-4f3574d97fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:57.907105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fe6ba2-78a7-417e-a5be-4f3574d97fa5 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:57.907864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17fe6ba2-78a7-417e-a5be-4f3574d97fa5 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:57.909585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17fe6ba2-78a7-417e-a5be-4f3574d97fa5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-17fe6ba2-78a7-417e-a5be-4f3574d97fa5 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:57.919454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:57.920521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 26 01:23:57.921469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:23:57.924315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 447/898] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:23:57 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:23:57.936396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fe6ba2-78a7-417e-a5be-4f3574d97fa5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:57.937393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:58.002785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d59cc0-2b26-4c0f-80ba-ab66d4a6afc5 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:58.030891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fe6ba2-78a7-417e-a5be-4f3574d97fa5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:58.043341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8281d3-d053-4d19-af25-1b2bc0f56fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:58.044453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8281d3-d053-4d19-af25-1b2bc0f56fe8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.044738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8281d3-d053-4d19-af25-1b2bc0f56fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.045010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8281d3-d053-4d19-af25-1b2bc0f56fe8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.053921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8281d3-d053-4d19-af25-1b2bc0f56fe8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:58.056148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8281d3-d053-4d19-af25-1b2bc0f56fe8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:58.061782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-7f8281d3-d053-4d19-af25-1b2bc0f56fe8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:23:58.067630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:58.068894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:58.069799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:23:58.069799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:23:58.069799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:23:58.069799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:23:58.069799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:23:58.069799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:23:58.069799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:23:58.069799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:58.073625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 452/899] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:23:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:58.073733 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546f2a88-1614-4de1-957b-80ae205258e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:58.074238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546f2a88-1614-4de1-957b-80ae205258e1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.074543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546f2a88-1614-4de1-957b-80ae205258e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.074821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546f2a88-1614-4de1-957b-80ae205258e1 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.074974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d926b5-b92c-47ff-9c8f-1499f5ea1afe None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.075247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d926b5-b92c-47ff-9c8f-1499f5ea1afe None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.075457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546f2a88-1614-4de1-957b-80ae205258e1 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:58.075560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d926b5-b92c-47ff-9c8f-1499f5ea1afe None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.075813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546f2a88-1614-4de1-957b-80ae205258e1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:58.076922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 453/900] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:58.077552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d926b5-b92c-47ff-9c8f-1499f5ea1afe None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:58.082213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546f2a88-1614-4de1-957b-80ae205258e1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:58.087519 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 448/901] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:23:57 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:23:58.088767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d6a34c-d419-4155-b817-87e906ba8d9d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.089054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d6a34c-d419-4155-b817-87e906ba8d9d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.089328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d6a34c-d419-4155-b817-87e906ba8d9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.108465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-98d6a34c-d419-4155-b817-87e906ba8d9d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:58.109767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-98d6a34c-d419-4155-b817-87e906ba8d9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:58.180982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 449/902] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:58 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:23:58.189558 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d926b5-b92c-47ff-9c8f-1499f5ea1afe None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:58.193705 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d6a34c-d419-4155-b817-87e906ba8d9d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:58.216833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:58.228458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edbf2aa-0424-4415-9c7b-45cd4a7a2ac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:58.229069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edbf2aa-0424-4415-9c7b-45cd4a7a2ac8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.229348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edbf2aa-0424-4415-9c7b-45cd4a7a2ac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.229621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edbf2aa-0424-4415-9c7b-45cd4a7a2ac8 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.230419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edbf2aa-0424-4415-9c7b-45cd4a7a2ac8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:58.230729 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edbf2aa-0424-4415-9c7b-45cd4a7a2ac8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:58.232329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edbf2aa-0424-4415-9c7b-45cd4a7a2ac8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:58.250245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 454/903] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:23:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:58.251535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edbf2aa-0424-4415-9c7b-45cd4a7a2ac8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:58.324517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d6a34c-d419-4155-b817-87e906ba8d9d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:58.356504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:58.360093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edbf2aa-0424-4415-9c7b-45cd4a7a2ac8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:58.376594 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62010d4f-25ac-47bf-ab16-ed160d10dc72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:58.377451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62010d4f-25ac-47bf-ab16-ed160d10dc72 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.377728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62010d4f-25ac-47bf-ab16-ed160d10dc72 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.378249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62010d4f-25ac-47bf-ab16-ed160d10dc72 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.379256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62010d4f-25ac-47bf-ab16-ed160d10dc72 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:58.379568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62010d4f-25ac-47bf-ab16-ed160d10dc72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:58.380964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62010d4f-25ac-47bf-ab16-ed160d10dc72 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:58.388388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 450/904] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:23:58 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:58.393918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62010d4f-25ac-47bf-ab16-ed160d10dc72 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:58.401564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76bae9c-dda4-4e8b-a44b-3fb5035777aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:58.402216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76bae9c-dda4-4e8b-a44b-3fb5035777aa None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.402498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76bae9c-dda4-4e8b-a44b-3fb5035777aa None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.402770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76bae9c-dda4-4e8b-a44b-3fb5035777aa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.403601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76bae9c-dda4-4e8b-a44b-3fb5035777aa None admin] RBAC: Adding query filter params (role.id=fe72ae0ebcab44b69a769caf591d6eda, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=50a8d0782b6f47d1aec027c6e15018a8) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:58.403970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76bae9c-dda4-4e8b-a44b-3fb5035777aa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:58.405579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76bae9c-dda4-4e8b-a44b-3fb5035777aa None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:58.420568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 455/905] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:23:58 2022] GET /identity/v3/role_assignments?user.id=50a8d0782b6f47d1aec027c6e15018a8&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=fe72ae0ebcab44b69a769caf591d6eda => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:58.463275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76bae9c-dda4-4e8b-a44b-3fb5035777aa None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:58.492695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:58.509349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62010d4f-25ac-47bf-ab16-ed160d10dc72 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:58.554454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a961881f-58d0-49c6-b6b0-31f831361eeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:58.555594 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a961881f-58d0-49c6-b6b0-31f831361eeb None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.555960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a961881f-58d0-49c6-b6b0-31f831361eeb None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.556236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a961881f-58d0-49c6-b6b0-31f831361eeb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.557003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a961881f-58d0-49c6-b6b0-31f831361eeb None admin] RBAC: Adding query filter params (interface=public, service_id=930278dda465435c838b8e7cb6f94bd0, region_id=RegionOne) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:58.557311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a961881f-58d0-49c6-b6b0-31f831361eeb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:58.558734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a961881f-58d0-49c6-b6b0-31f831361eeb None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:58.572434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 451/906] 10.222.0.25 () {56 vars in 1251 bytes} [Mon Sep 26 01:23:58 2022] GET /identity/v3/endpoints?service_id=930278dda465435c838b8e7cb6f94bd0&interface=public®ion_id=RegionOne => generated 194 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:58.576292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a961881f-58d0-49c6-b6b0-31f831361eeb None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:58.596652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76bae9c-dda4-4e8b-a44b-3fb5035777aa None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:58.638494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:58.643067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cfa42a-7c4e-43dc-a5ae-d392b451c2bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:58.644026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cfa42a-7c4e-43dc-a5ae-d392b451c2bb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.644655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cfa42a-7c4e-43dc-a5ae-d392b451c2bb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.645077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cfa42a-7c4e-43dc-a5ae-d392b451c2bb None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.646289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cfa42a-7c4e-43dc-a5ae-d392b451c2bb None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:58.649406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cfa42a-7c4e-43dc-a5ae-d392b451c2bb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-78cfa42a-7c4e-43dc-a5ae-d392b451c2bb None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:23:58.660273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:23:58.661478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 26 01:23:58.662605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:23:58.666522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 456/907] 10.222.0.25 () {56 vars in 1132 bytes} [Mon Sep 26 01:23:58 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:23:58.681344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cfa42a-7c4e-43dc-a5ae-d392b451c2bb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:58.705115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a961881f-58d0-49c6-b6b0-31f831361eeb None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:58.742593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ce02c6-b44a-4cde-9a9d-80f3f650de2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:58.743530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ce02c6-b44a-4cde-9a9d-80f3f650de2a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.744064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ce02c6-b44a-4cde-9a9d-80f3f650de2a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.744522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ce02c6-b44a-4cde-9a9d-80f3f650de2a None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.745272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ce02c6-b44a-4cde-9a9d-80f3f650de2a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:58.746807 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ce02c6-b44a-4cde-9a9d-80f3f650de2a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:58.753907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 452/908] 10.222.0.25 () {56 vars in 1087 bytes} [Mon Sep 26 01:23:58 2022] GET /identity/v3/services => generated 1497 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:23:58.792239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:58.806949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cfa42a-7c4e-43dc-a5ae-d392b451c2bb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:58.846313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37809e8-64df-4be4-ae1b-96ddb3c8fc4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:58.847083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37809e8-64df-4be4-ae1b-96ddb3c8fc4e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:58.847524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37809e8-64df-4be4-ae1b-96ddb3c8fc4e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:58.847898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37809e8-64df-4be4-ae1b-96ddb3c8fc4e None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:58.848681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37809e8-64df-4be4-ae1b-96ddb3c8fc4e None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:58.848990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37809e8-64df-4be4-ae1b-96ddb3c8fc4e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:58.850403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37809e8-64df-4be4-ae1b-96ddb3c8fc4e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:58.857208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 457/909] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:23:58 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:58.862923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ce02c6-b44a-4cde-9a9d-80f3f650de2a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:58.974496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ce02c6-b44a-4cde-9a9d-80f3f650de2a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:58.988792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:59.012636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffbbf70-2ee6-4e2e-a643-67ce9cb796e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:59.013577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffbbf70-2ee6-4e2e-a643-67ce9cb796e4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:59.014057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffbbf70-2ee6-4e2e-a643-67ce9cb796e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:59.014640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffbbf70-2ee6-4e2e-a643-67ce9cb796e4 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:59.015470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffbbf70-2ee6-4e2e-a643-67ce9cb796e4 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:59.016032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffbbf70-2ee6-4e2e-a643-67ce9cb796e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:59.017698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffbbf70-2ee6-4e2e-a643-67ce9cb796e4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:59.023889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 453/910] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:23:58 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:59.030068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37809e8-64df-4be4-ae1b-96ddb3c8fc4e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:59.136283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37809e8-64df-4be4-ae1b-96ddb3c8fc4e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:59.172529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4f5bee-3ba3-4e6c-bcaa-505ead4682b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:59.173451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f5bee-3ba3-4e6c-bcaa-505ead4682b7 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:59.173902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f5bee-3ba3-4e6c-bcaa-505ead4682b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:59.174346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f5bee-3ba3-4e6c-bcaa-505ead4682b7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:59.175189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4f5bee-3ba3-4e6c-bcaa-505ead4682b7 None admin] RBAC: Adding query filter params (interface=public, service_id=60c321b4ff2b4990a0685a6dfcb3df22, region_id=RegionOne) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:23:59.175643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4f5bee-3ba3-4e6c-bcaa-505ead4682b7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:59.177173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4f5bee-3ba3-4e6c-bcaa-505ead4682b7 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:59.183968 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 458/911] 10.222.0.25 () {56 vars in 1251 bytes} [Mon Sep 26 01:23:59 2022] GET /identity/v3/endpoints?service_id=60c321b4ff2b4990a0685a6dfcb3df22&interface=public®ion_id=RegionOne => generated 194 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:23:59.192867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffbbf70-2ee6-4e2e-a643-67ce9cb796e4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:23:59.300837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffbbf70-2ee6-4e2e-a643-67ce9cb796e4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:23:59.336486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a884db-d56b-4398-a012-0d42e99d3e3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:23:59.337096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a884db-d56b-4398-a012-0d42e99d3e3f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:59.337374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a884db-d56b-4398-a012-0d42e99d3e3f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:59.337646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a884db-d56b-4398-a012-0d42e99d3e3f None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:59.338255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a884db-d56b-4398-a012-0d42e99d3e3f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:23:59.339636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a884db-d56b-4398-a012-0d42e99d3e3f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:23:59.346151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 454/912] 10.222.0.25 () {56 vars in 1087 bytes} [Mon Sep 26 01:23:59 2022] GET /identity/v3/services => generated 1497 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:23:59.621045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894e47cf-b4c0-46e2-a8c7-30fd7c79fe16 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:59.621534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894e47cf-b4c0-46e2-a8c7-30fd7c79fe16 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:59.621534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894e47cf-b4c0-46e2-a8c7-30fd7c79fe16 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:59.623054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 459/913] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:23:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:23:59.628533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b7424e-bd87-4c74-a7a6-d01a9a40388f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:23:59.628954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b7424e-bd87-4c74-a7a6-d01a9a40388f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:23:59.629127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b7424e-bd87-4c74-a7a6-d01a9a40388f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:23:59.648178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-35b7424e-bd87-4c74-a7a6-d01a9a40388f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:23:59.649560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-35b7424e-bd87-4c74-a7a6-d01a9a40388f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:23:59.685921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:23:59.722881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 455/914] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:23:59 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:00.001818 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894e47cf-b4c0-46e2-a8c7-30fd7c79fe16 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:00.020871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:00.114097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894e47cf-b4c0-46e2-a8c7-30fd7c79fe16 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:00.150477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e30509-1f00-4473-b6ee-6ba0a230b399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:00.151099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e30509-1f00-4473-b6ee-6ba0a230b399 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:00.151375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e30509-1f00-4473-b6ee-6ba0a230b399 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:00.151646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e30509-1f00-4473-b6ee-6ba0a230b399 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:00.152227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e30509-1f00-4473-b6ee-6ba0a230b399 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:00.153608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e30509-1f00-4473-b6ee-6ba0a230b399 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-e8e30509-1f00-4473-b6ee-6ba0a230b399 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:00.162484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:00.163465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:00.165499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 460/915] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:24:00 2022] GET /identity/v3/services/object-store => generated 93 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:00.170262 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b7424e-bd87-4c74-a7a6-d01a9a40388f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:00.274216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b7424e-bd87-4c74-a7a6-d01a9a40388f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:00.312995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7873647c-4b0f-4081-b833-bf6c3168ea4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:00.313604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7873647c-4b0f-4081-b833-bf6c3168ea4d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:00.313871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7873647c-4b0f-4081-b833-bf6c3168ea4d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:00.314146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7873647c-4b0f-4081-b833-bf6c3168ea4d None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:00.314779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7873647c-4b0f-4081-b833-bf6c3168ea4d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:00.315089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7873647c-4b0f-4081-b833-bf6c3168ea4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:00.316487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7873647c-4b0f-4081-b833-bf6c3168ea4d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:00.322578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 456/916] 10.222.0.25 () {56 vars in 1122 bytes} [Mon Sep 26 01:24:00 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:00.328253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e30509-1f00-4473-b6ee-6ba0a230b399 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:00.420169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:00.440958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e30509-1f00-4473-b6ee-6ba0a230b399 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:00.477621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93db056c-3fb6-4fa8-afc2-1c23717a0904 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:00.478230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93db056c-3fb6-4fa8-afc2-1c23717a0904 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:00.478505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93db056c-3fb6-4fa8-afc2-1c23717a0904 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:00.478770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93db056c-3fb6-4fa8-afc2-1c23717a0904 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:00.479397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93db056c-3fb6-4fa8-afc2-1c23717a0904 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:00.479696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93db056c-3fb6-4fa8-afc2-1c23717a0904 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:00.481075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93db056c-3fb6-4fa8-afc2-1c23717a0904 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:00.486385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 461/917] 10.222.0.25 () {56 vars in 1122 bytes} [Mon Sep 26 01:24:00 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:00.492773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7873647c-4b0f-4081-b833-bf6c3168ea4d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:00.529120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:00.603457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7873647c-4b0f-4081-b833-bf6c3168ea4d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:00.646362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df90b4f-e7f1-47af-a21a-b7e4d0db57f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:00.647956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df90b4f-e7f1-47af-a21a-b7e4d0db57f6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:00.649067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df90b4f-e7f1-47af-a21a-b7e4d0db57f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:00.649568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df90b4f-e7f1-47af-a21a-b7e4d0db57f6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:00.650624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df90b4f-e7f1-47af-a21a-b7e4d0db57f6 None admin] RBAC: Adding query filter params (interface=public, service_id=a46c30fe636844b7be5e8d1382b68d48, region_id=RegionOne) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:00.651137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df90b4f-e7f1-47af-a21a-b7e4d0db57f6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:00.652747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df90b4f-e7f1-47af-a21a-b7e4d0db57f6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:00.658975 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 457/918] 10.222.0.25 () {56 vars in 1251 bytes} [Mon Sep 26 01:24:00 2022] GET /identity/v3/endpoints?service_id=a46c30fe636844b7be5e8d1382b68d48&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:00.665641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93db056c-3fb6-4fa8-afc2-1c23717a0904 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:00.738700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e38d3a-09b9-4993-a4e9-5c815da741ba None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:00.739036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e38d3a-09b9-4993-a4e9-5c815da741ba None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:00.739234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e38d3a-09b9-4993-a4e9-5c815da741ba None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:00.740827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 458/919] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:24:00.746636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c180f06-79c7-4c68-bb6f-20904cca8087 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:00.747232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c180f06-79c7-4c68-bb6f-20904cca8087 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:00.747819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c180f06-79c7-4c68-bb6f-20904cca8087 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:00.766962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0c180f06-79c7-4c68-bb6f-20904cca8087 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:00.768612 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0c180f06-79c7-4c68-bb6f-20904cca8087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:00.779856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93db056c-3fb6-4fa8-afc2-1c23717a0904 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:00.816030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5e3ffd-9b19-4c7a-ae12-c495f9adfa9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:00.816662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e3ffd-9b19-4c7a-ae12-c495f9adfa9d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:00.816942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e3ffd-9b19-4c7a-ae12-c495f9adfa9d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:00.817217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e3ffd-9b19-4c7a-ae12-c495f9adfa9d None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:00.817822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5e3ffd-9b19-4c7a-ae12-c495f9adfa9d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:00.819186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5e3ffd-9b19-4c7a-ae12-c495f9adfa9d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:00.826210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 462/920] 10.222.0.25 () {56 vars in 1087 bytes} [Mon Sep 26 01:24:00 2022] GET /identity/v3/services => generated 1497 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:24:00.858392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 459/921] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:00 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:01.051613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b1a4ab-d820-4707-8a10-f5bf4bd56de0 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:01.051898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b1a4ab-d820-4707-8a10-f5bf4bd56de0 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:01.052174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b1a4ab-d820-4707-8a10-f5bf4bd56de0 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:01.053795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 463/922] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:01.060278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c59ae8b-282f-4119-a7c6-a018563153be None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:01.060605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c59ae8b-282f-4119-a7c6-a018563153be None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:01.060887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c59ae8b-282f-4119-a7c6-a018563153be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:01.080107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7c59ae8b-282f-4119-a7c6-a018563153be None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:01.081327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7c59ae8b-282f-4119-a7c6-a018563153be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:01.105498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:01.142522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b1a4ab-d820-4707-8a10-f5bf4bd56de0 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:01.154221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 460/923] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:01 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:24:01.156850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:01.254463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b1a4ab-d820-4707-8a10-f5bf4bd56de0 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:01.290203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec59312-c64a-4893-abf1-86e0df08c0e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:01.291249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec59312-c64a-4893-abf1-86e0df08c0e2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:01.291758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec59312-c64a-4893-abf1-86e0df08c0e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:01.292232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec59312-c64a-4893-abf1-86e0df08c0e2 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:01.303285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec59312-c64a-4893-abf1-86e0df08c0e2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-dec59312-c64a-4893-abf1-86e0df08c0e2 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:01.308901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:01.309848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:24:01.310715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:01.313458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 464/924] 10.222.0.25 () {56 vars in 1120 bytes} [Mon Sep 26 01:24:01 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:01.320450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c59ae8b-282f-4119-a7c6-a018563153be None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:01.427771 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c59ae8b-282f-4119-a7c6-a018563153be None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:01.440145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec59312-c64a-4893-abf1-86e0df08c0e2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:01.471553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5f0537-bc87-4486-926b-22ccc11a5361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:01.472320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5f0537-bc87-4486-926b-22ccc11a5361 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:01.472721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5f0537-bc87-4486-926b-22ccc11a5361 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:01.472994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5f0537-bc87-4486-926b-22ccc11a5361 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:01.473724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5f0537-bc87-4486-926b-22ccc11a5361 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:01.474034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5f0537-bc87-4486-926b-22ccc11a5361 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:01.475903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5f0537-bc87-4486-926b-22ccc11a5361 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:01.490398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 461/925] 10.222.0.25 () {56 vars in 1115 bytes} [Mon Sep 26 01:24:01 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:01.493021 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5f0537-bc87-4486-926b-22ccc11a5361 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:01.517433 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:01.520948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:01.566570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec59312-c64a-4893-abf1-86e0df08c0e2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:01.621258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fd886c-38c6-4bb0-8c00-0be95a30b76d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:01.622001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fd886c-38c6-4bb0-8c00-0be95a30b76d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:01.622288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fd886c-38c6-4bb0-8c00-0be95a30b76d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:01.622560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fd886c-38c6-4bb0-8c00-0be95a30b76d None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:01.623152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fd886c-38c6-4bb0-8c00-0be95a30b76d None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:01.625317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fd886c-38c6-4bb0-8c00-0be95a30b76d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:01.631489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5f0537-bc87-4486-926b-22ccc11a5361 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-89fd886c-38c6-4bb0-8c00-0be95a30b76d None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:01.637594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:01.638627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 26 01:24:01.639470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:01.644091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 465/926] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:24:01 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:01.645381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fd886c-38c6-4bb0-8c00-0be95a30b76d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:01.679477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab59a341-0fb9-4a0a-b7d0-1d65f9e278c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:01.681385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab59a341-0fb9-4a0a-b7d0-1d65f9e278c2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:01.681839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab59a341-0fb9-4a0a-b7d0-1d65f9e278c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:01.682258 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab59a341-0fb9-4a0a-b7d0-1d65f9e278c2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:01.701102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab59a341-0fb9-4a0a-b7d0-1d65f9e278c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:01.702888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab59a341-0fb9-4a0a-b7d0-1d65f9e278c2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ab59a341-0fb9-4a0a-b7d0-1d65f9e278c2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:01.718056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:01.719048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:01.719962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:01.719962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:01.719962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:01.719962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 462/927] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:01.720646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a105cf59-ccec-4c74-a2ef-d24aef8b22d2 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:01.720646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a105cf59-ccec-4c74-a2ef-d24aef8b22d2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:01.720929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a105cf59-ccec-4c74-a2ef-d24aef8b22d2 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:01.724174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 463/928] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:01.724620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a105cf59-ccec-4c74-a2ef-d24aef8b22d2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:01.773221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fd886c-38c6-4bb0-8c00-0be95a30b76d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:01.805073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:01.805750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:01.812678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188da158-b21c-42c0-be59-9a846ba708ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:01.813291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188da158-b21c-42c0-be59-9a846ba708ba None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:01.813626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188da158-b21c-42c0-be59-9a846ba708ba None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:01.813900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188da158-b21c-42c0-be59-9a846ba708ba None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:01.817239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188da158-b21c-42c0-be59-9a846ba708ba None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:01.817414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188da158-b21c-42c0-be59-9a846ba708ba None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:01.818777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188da158-b21c-42c0-be59-9a846ba708ba None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:01.826980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 466/929] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:24:01 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:01.828486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eecbce-1667-4793-b8bf-9007f03e7cd7 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:01.828835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eecbce-1667-4793-b8bf-9007f03e7cd7 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:01.829177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eecbce-1667-4793-b8bf-9007f03e7cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:01.848209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-61eecbce-1667-4793-b8bf-9007f03e7cd7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:01.849512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-61eecbce-1667-4793-b8bf-9007f03e7cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:01.850550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a105cf59-ccec-4c74-a2ef-d24aef8b22d2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:01.888906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6a6052-880b-4984-88c0-c1f84eb68199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:01.889512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6a6052-880b-4984-88c0-c1f84eb68199 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:01.889790 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6a6052-880b-4984-88c0-c1f84eb68199 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:01.890067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6a6052-880b-4984-88c0-c1f84eb68199 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:01.890702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6a6052-880b-4984-88c0-c1f84eb68199 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:01.891006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6a6052-880b-4984-88c0-c1f84eb68199 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:01.892386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6a6052-880b-4984-88c0-c1f84eb68199 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:01.909260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 464/930] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:01.909260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6a6052-880b-4984-88c0-c1f84eb68199 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:01.930389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 467/931] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:01 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:01.931672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eecbce-1667-4793-b8bf-9007f03e7cd7 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:02.012656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6a6052-880b-4984-88c0-c1f84eb68199 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.054166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:02.060490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5640b77c-cecf-4849-a650-1046ec91152b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:02.061454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5640b77c-cecf-4849-a650-1046ec91152b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:02.062127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5640b77c-cecf-4849-a650-1046ec91152b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:02.062268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5640b77c-cecf-4849-a650-1046ec91152b None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:02.063062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5640b77c-cecf-4849-a650-1046ec91152b None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:02.063422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5640b77c-cecf-4849-a650-1046ec91152b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:02.065037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5640b77c-cecf-4849-a650-1046ec91152b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:02.070241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eecbce-1667-4793-b8bf-9007f03e7cd7 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.071018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 465/932] 10.222.0.25 () {56 vars in 1114 bytes} [Mon Sep 26 01:24:01 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:02.076673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5640b77c-cecf-4849-a650-1046ec91152b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:02.107821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:02.121119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31765ace-27ac-4a13-8fe1-a5efcbfc9783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:02.122554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31765ace-27ac-4a13-8fe1-a5efcbfc9783 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:02.122981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31765ace-27ac-4a13-8fe1-a5efcbfc9783 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:02.123418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31765ace-27ac-4a13-8fe1-a5efcbfc9783 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:02.133866 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31765ace-27ac-4a13-8fe1-a5efcbfc9783 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:02.135475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31765ace-27ac-4a13-8fe1-a5efcbfc9783 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-31765ace-27ac-4a13-8fe1-a5efcbfc9783 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:24:02.152755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:02.153815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:02.154764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:02.154764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:02.154764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:24:02.154764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:24:02.154764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:24:02.154764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:24:02.154764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:24:02.154764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:24:02.154764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:24:02.154764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:02.162570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 468/933] 10.222.0.25 () {56 vars in 1117 bytes} [Mon Sep 26 01:24:01 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:02.163576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31765ace-27ac-4a13-8fe1-a5efcbfc9783 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:02.189273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5640b77c-cecf-4849-a650-1046ec91152b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.229078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79da955-0869-4971-b882-f9b1d8255476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:02.229673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79da955-0869-4971-b882-f9b1d8255476 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:02.229948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79da955-0869-4971-b882-f9b1d8255476 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:02.230212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79da955-0869-4971-b882-f9b1d8255476 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:02.230865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79da955-0869-4971-b882-f9b1d8255476 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:02.232232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79da955-0869-4971-b882-f9b1d8255476 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:02.256387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 466/934] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:24:02 2022] POST /identity/v3/endpoints => generated 346 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:24:02.256387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79da955-0869-4971-b882-f9b1d8255476 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:02.293073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31765ace-27ac-4a13-8fe1-a5efcbfc9783 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.350749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9903ddf-2e53-4141-8d69-68a5b854d998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:02.352391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9903ddf-2e53-4141-8d69-68a5b854d998 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:02.352700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9903ddf-2e53-4141-8d69-68a5b854d998 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:02.352976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9903ddf-2e53-4141-8d69-68a5b854d998 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:02.354103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9903ddf-2e53-4141-8d69-68a5b854d998 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:02.354422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9903ddf-2e53-4141-8d69-68a5b854d998 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:02.356049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9903ddf-2e53-4141-8d69-68a5b854d998 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:02.376815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:02.384795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 469/935] 10.222.0.25 () {56 vars in 1149 bytes} [Mon Sep 26 01:24:02 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:02.387730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9903ddf-2e53-4141-8d69-68a5b854d998 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:02.406646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79da955-0869-4971-b882-f9b1d8255476 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.436324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:02.450560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd9716a-58c3-4ef2-aaeb-47c50d50b26c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:02.452071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd9716a-58c3-4ef2-aaeb-47c50d50b26c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:02.452531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd9716a-58c3-4ef2-aaeb-47c50d50b26c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:02.453211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd9716a-58c3-4ef2-aaeb-47c50d50b26c None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:02.453938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd9716a-58c3-4ef2-aaeb-47c50d50b26c None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:02.455869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd9716a-58c3-4ef2-aaeb-47c50d50b26c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-1bd9716a-58c3-4ef2-aaeb-47c50d50b26c None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:02.469340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:02.470515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:02.471793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 467/936] 10.222.0.25 () {56 vars in 1132 bytes} [Mon Sep 26 01:24:02 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:02.476892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd9716a-58c3-4ef2-aaeb-47c50d50b26c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:02.516833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9903ddf-2e53-4141-8d69-68a5b854d998 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.567805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1886a467-0587-4ada-99ce-9b45fa4dd5eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:02.569391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886a467-0587-4ada-99ce-9b45fa4dd5eb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:02.569822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886a467-0587-4ada-99ce-9b45fa4dd5eb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:02.570494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886a467-0587-4ada-99ce-9b45fa4dd5eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:02.584119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1886a467-0587-4ada-99ce-9b45fa4dd5eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:02.586677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1886a467-0587-4ada-99ce-9b45fa4dd5eb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-1886a467-0587-4ada-99ce-9b45fa4dd5eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:02.607563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:02.608763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:02.609863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:02.609863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:02.609863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:02.612426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 470/937] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:02.618916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd9716a-58c3-4ef2-aaeb-47c50d50b26c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.621403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1886a467-0587-4ada-99ce-9b45fa4dd5eb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:02.638295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:02.658723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a65c764-92f6-4930-b7a3-b2b89b5d4fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:02.659332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a65c764-92f6-4930-b7a3-b2b89b5d4fc5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:02.659629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a65c764-92f6-4930-b7a3-b2b89b5d4fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:02.659947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a65c764-92f6-4930-b7a3-b2b89b5d4fc5 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:02.660537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a65c764-92f6-4930-b7a3-b2b89b5d4fc5 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:02.660841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a65c764-92f6-4930-b7a3-b2b89b5d4fc5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:02.662332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a65c764-92f6-4930-b7a3-b2b89b5d4fc5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:02.673362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 468/938] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:24:02 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:02.674330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a65c764-92f6-4930-b7a3-b2b89b5d4fc5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:02.733177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:02.743999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1886a467-0587-4ada-99ce-9b45fa4dd5eb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.781895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8653e0d7-da38-421c-a7b5-ec029b405426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:02.782676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8653e0d7-da38-421c-a7b5-ec029b405426 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:02.783199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8653e0d7-da38-421c-a7b5-ec029b405426 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:02.783581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8653e0d7-da38-421c-a7b5-ec029b405426 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:02.784308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8653e0d7-da38-421c-a7b5-ec029b405426 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:02.784731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8653e0d7-da38-421c-a7b5-ec029b405426 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:02.786482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8653e0d7-da38-421c-a7b5-ec029b405426 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:02.789818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a65c764-92f6-4930-b7a3-b2b89b5d4fc5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.806300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 471/939] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:02.807443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8653e0d7-da38-421c-a7b5-ec029b405426 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:02.828274 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fe5f80-70fa-4504-93c5-4f66b6f9e0d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:02.829211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fe5f80-70fa-4504-93c5-4f66b6f9e0d0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:02.829660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fe5f80-70fa-4504-93c5-4f66b6f9e0d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:02.830095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fe5f80-70fa-4504-93c5-4f66b6f9e0d0 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:02.830875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fe5f80-70fa-4504-93c5-4f66b6f9e0d0 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:02.831334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fe5f80-70fa-4504-93c5-4f66b6f9e0d0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:02.833234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fe5f80-70fa-4504-93c5-4f66b6f9e0d0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:02.843985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 469/940] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:24:02 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:02.845073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fe5f80-70fa-4504-93c5-4f66b6f9e0d0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:02.916980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8653e0d7-da38-421c-a7b5-ec029b405426 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.936773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:02.955286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fe5f80-70fa-4504-93c5-4f66b6f9e0d0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:02.963442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a850b144-c571-4f8a-ad3f-d7b4b73c107d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:02.964257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a850b144-c571-4f8a-ad3f-d7b4b73c107d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:02.964664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a850b144-c571-4f8a-ad3f-d7b4b73c107d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:02.965036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a850b144-c571-4f8a-ad3f-d7b4b73c107d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:02.973404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a850b144-c571-4f8a-ad3f-d7b4b73c107d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:02.975174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a850b144-c571-4f8a-ad3f-d7b4b73c107d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-a850b144-c571-4f8a-ad3f-d7b4b73c107d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:24:02.988587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:02.989614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:02.990515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:24:02.990515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:24:02.990515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:24:02.990515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:24:02.990515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:24:02.990515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:24:02.990515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:02.990515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:02.990515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 472/941] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:24:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:02.996691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a850b144-c571-4f8a-ad3f-d7b4b73c107d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:03.000206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bd57e7-14bf-4c8e-b3ea-9c92c7b4652b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:03.001130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bd57e7-14bf-4c8e-b3ea-9c92c7b4652b None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:03.001590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bd57e7-14bf-4c8e-b3ea-9c92c7b4652b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:03.002057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bd57e7-14bf-4c8e-b3ea-9c92c7b4652b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:03.002900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bd57e7-14bf-4c8e-b3ea-9c92c7b4652b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:03.004483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bd57e7-14bf-4c8e-b3ea-9c92c7b4652b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:03.012661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:03.031224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 470/942] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:24:02 2022] POST /identity/v3/endpoints => generated 347 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:24:03.077850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb69984-34f0-486c-b83d-9b4e96fae940 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:03.078132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb69984-34f0-486c-b83d-9b4e96fae940 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:03.078413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb69984-34f0-486c-b83d-9b4e96fae940 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:03.080062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 471/943] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:24:03.086467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d2db92-3730-40b3-b2e8-db14c846796c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:03.086836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d2db92-3730-40b3-b2e8-db14c846796c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:03.087019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d2db92-3730-40b3-b2e8-db14c846796c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:03.107553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f4d2db92-3730-40b3-b2e8-db14c846796c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:03.109259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f4d2db92-3730-40b3-b2e8-db14c846796c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:03.123500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a850b144-c571-4f8a-ad3f-d7b4b73c107d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:03.169182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab55bff-c243-4909-b7c6-1bed3221f5ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:03.170128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab55bff-c243-4909-b7c6-1bed3221f5ed None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:03.170624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab55bff-c243-4909-b7c6-1bed3221f5ed None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:03.171068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab55bff-c243-4909-b7c6-1bed3221f5ed None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:03.171998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab55bff-c243-4909-b7c6-1bed3221f5ed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:03.172580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab55bff-c243-4909-b7c6-1bed3221f5ed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:03.174220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab55bff-c243-4909-b7c6-1bed3221f5ed None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:03.192680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 473/944] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:24:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:03.194590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab55bff-c243-4909-b7c6-1bed3221f5ed None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:03.209167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 472/945] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:03 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:24:03.274475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:03.309731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab55bff-c243-4909-b7c6-1bed3221f5ed None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:03.345922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d21048b-6d39-4381-9c99-c20a9508e7e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:03.346862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d21048b-6d39-4381-9c99-c20a9508e7e6 None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:03.347319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d21048b-6d39-4381-9c99-c20a9508e7e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:03.347762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d21048b-6d39-4381-9c99-c20a9508e7e6 None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/50a8d0782b6f47d1aec027c6e15018a8/roles/fe72ae0ebcab44b69a769caf591d6eda` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:03.350264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d21048b-6d39-4381-9c99-c20a9508e7e6 None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=50a8d0782b6f47d1aec027c6e15018a8, role_id=fe72ae0ebcab44b69a769caf591d6eda)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:03.352563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d21048b-6d39-4381-9c99-c20a9508e7e6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:03.370324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 474/946] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:24:03 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/50a8d0782b6f47d1aec027c6e15018a8/roles/fe72ae0ebcab44b69a769caf591d6eda => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:24:03.488869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d2db92-3730-40b3-b2e8-db14c846796c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:03.522419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:03.600518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d2db92-3730-40b3-b2e8-db14c846796c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:03.637710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0438d8d7-a00f-4b61-9870-ee323d1df490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:03.638345 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0438d8d7-a00f-4b61-9870-ee323d1df490 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:03.638625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0438d8d7-a00f-4b61-9870-ee323d1df490 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:03.638895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0438d8d7-a00f-4b61-9870-ee323d1df490 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:03.639480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0438d8d7-a00f-4b61-9870-ee323d1df490 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:03.640894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0438d8d7-a00f-4b61-9870-ee323d1df490 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0438d8d7-a00f-4b61-9870-ee323d1df490 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:03.650485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:03.651515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:03.652410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 473/947] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:24:03 2022] GET /identity/v3/services/object-store => generated 93 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:03.659829 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d21048b-6d39-4381-9c99-c20a9508e7e6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:03.769252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d21048b-6d39-4381-9c99-c20a9508e7e6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:03.797135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:03.806862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac8a9f1-6ba9-4820-9af9-096289ab8536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:03.807469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac8a9f1-6ba9-4820-9af9-096289ab8536 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:03.807757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac8a9f1-6ba9-4820-9af9-096289ab8536 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:03.808008 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac8a9f1-6ba9-4820-9af9-096289ab8536 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:03.808738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac8a9f1-6ba9-4820-9af9-096289ab8536 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:03.809109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac8a9f1-6ba9-4820-9af9-096289ab8536 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:03.810594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac8a9f1-6ba9-4820-9af9-096289ab8536 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:03.815907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 475/948] 10.222.0.25 () {56 vars in 1122 bytes} [Mon Sep 26 01:24:03 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:03.822687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0438d8d7-a00f-4b61-9870-ee323d1df490 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:03.922473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:03.929673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0438d8d7-a00f-4b61-9870-ee323d1df490 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:03.966818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddd76ab-8c58-4d9e-90eb-252a0a502600 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:03.967425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddd76ab-8c58-4d9e-90eb-252a0a502600 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:03.967705 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddd76ab-8c58-4d9e-90eb-252a0a502600 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:03.967973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddd76ab-8c58-4d9e-90eb-252a0a502600 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:03.968636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddd76ab-8c58-4d9e-90eb-252a0a502600 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:03.968942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddd76ab-8c58-4d9e-90eb-252a0a502600 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:03.970310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddd76ab-8c58-4d9e-90eb-252a0a502600 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:03.975686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 474/949] 10.222.0.25 () {56 vars in 1122 bytes} [Mon Sep 26 01:24:03 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:03.982167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac8a9f1-6ba9-4820-9af9-096289ab8536 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:04.091138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac8a9f1-6ba9-4820-9af9-096289ab8536 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:04.126539 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac02d7b-7605-4b61-9f50-00e949e2eef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:04.127146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac02d7b-7605-4b61-9f50-00e949e2eef7 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:04.127425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac02d7b-7605-4b61-9f50-00e949e2eef7 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:04.127702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac02d7b-7605-4b61-9f50-00e949e2eef7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:04.128533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac02d7b-7605-4b61-9f50-00e949e2eef7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:04.129913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac02d7b-7605-4b61-9f50-00e949e2eef7 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:04.154406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 476/950] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:24:03 2022] POST /identity/v3/endpoints => generated 349 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:24:04.605269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39086ea-a003-4cd4-b515-0ade075f7e61 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:04.605650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39086ea-a003-4cd4-b515-0ade075f7e61 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:04.605858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39086ea-a003-4cd4-b515-0ade075f7e61 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:04.607420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 475/951] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:04.612589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50988f09-cf28-416f-9e3d-ec7fecc4d812 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:04.612879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50988f09-cf28-416f-9e3d-ec7fecc4d812 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:04.613358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50988f09-cf28-416f-9e3d-ec7fecc4d812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:04.633313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-50988f09-cf28-416f-9e3d-ec7fecc4d812 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:04.634521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-50988f09-cf28-416f-9e3d-ec7fecc4d812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:04.680845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:04.711100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 477/952] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:04 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:04.986872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39086ea-a003-4cd4-b515-0ade075f7e61 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:05.100011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39086ea-a003-4cd4-b515-0ade075f7e61 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:05.138379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:05.139366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:05.139849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:05.140353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:05.141240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:05.142924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:05.145731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: WARNING keystone.server.flask.common [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 26 01:24:05.166140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.sql.core [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 26 01:24:05.166140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 26 01:24:05.166140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [parameters: {'id': '5609ba74086545d5a5add18423210adc', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 26 01:24:05.166140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81423) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 5609ba74086545d5a5add18423210adc.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 5609ba74086545d5a5add18423210adc. Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result = self._query(query) Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application conn.query(q) Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 26 01:24:05.173930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result.read() Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 26 01:24:05.174768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application next(self.gen) Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application yield resource Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application next(self.gen) Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.session.rollback() Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.force_reraise() Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise self.value Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application session.commit() Sep 26 01:24:05.175725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._prepare_impl() Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.session.flush() Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._flush(objects) Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application compat.raise_( Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application flush_context.execute() Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 26 01:24:05.176666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rec.execute(self) Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 26 01:24:05.177759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result = self._query(query) Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application conn.query(q) Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result.read() Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 26 01:24:05.178808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [parameters: {'id': '5609ba74086545d5a5add18423210adc', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:05.180558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:05.181487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:05.182456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:05.182456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 26 01:24:05.182456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 26 01:24:05.182456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:05.182456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:05.182456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 26 01:24:05.182456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 26 01:24:05.182456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 5609ba74086545d5a5add18423210adc. Sep 26 01:24:05.182456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:05.184924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 476/953] 10.222.0.25 () {60 vars in 1140 bytes} [Mon Sep 26 01:24:04 2022] POST /identity/v3/projects => generated 262 bytes in 198 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:05.192442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50988f09-cf28-416f-9e3d-ec7fecc4d812 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:05.301133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50988f09-cf28-416f-9e3d-ec7fecc4d812 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:05.337855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5625542b-6977-42e7-bf16-d196c80ad151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:05.338491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5625542b-6977-42e7-bf16-d196c80ad151 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:05.338771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5625542b-6977-42e7-bf16-d196c80ad151 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:05.339045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5625542b-6977-42e7-bf16-d196c80ad151 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:05.345990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5625542b-6977-42e7-bf16-d196c80ad151 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:05.347525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5625542b-6977-42e7-bf16-d196c80ad151 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-5625542b-6977-42e7-bf16-d196c80ad151 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:24:05.359557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:05.361836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:05.364063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:24:05.364063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:24:05.364063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:24:05.364063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:24:05.364063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:24:05.364063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:24:05.364063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:05.364063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:05.364063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 478/954] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:24:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:05.369868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:05.397095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:05.399623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b834171-b8bf-4e80-a9bf-253d6917a868 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:05.399915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b834171-b8bf-4e80-a9bf-253d6917a868 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:05.400190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b834171-b8bf-4e80-a9bf-253d6917a868 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:05.401693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 479/955] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:05.407676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032536d1-4185-464b-9615-687341958fce None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:05.408213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032536d1-4185-464b-9615-687341958fce None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:05.408724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032536d1-4185-464b-9615-687341958fce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:05.432327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-032536d1-4185-464b-9615-687341958fce None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:05.433861 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-032536d1-4185-464b-9615-687341958fce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:05.464649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:05.493839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ca97db-be9a-45fb-8003-b280a8e7fc15 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:05.528381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 480/956] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:05 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:05.544782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6615601c-4d08-4e34-9e82-849259ce9714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:05.545778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6615601c-4d08-4e34-9e82-849259ce9714 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:05.546211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6615601c-4d08-4e34-9e82-849259ce9714 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:05.546633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6615601c-4d08-4e34-9e82-849259ce9714 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:05.547494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6615601c-4d08-4e34-9e82-849259ce9714 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:05.547971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6615601c-4d08-4e34-9e82-849259ce9714 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:05.549872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6615601c-4d08-4e34-9e82-849259ce9714 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:05.575120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 477/957] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:24:05 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:05.663780 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec43a222-dfcb-4f1a-befe-c8055a60b64e None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:05.664104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec43a222-dfcb-4f1a-befe-c8055a60b64e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:05.664396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec43a222-dfcb-4f1a-befe-c8055a60b64e None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:05.665967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 481/958] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:05.671620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419ce970-edac-449b-a1c7-b5592db86b7b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:05.672164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419ce970-edac-449b-a1c7-b5592db86b7b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:05.672662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419ce970-edac-449b-a1c7-b5592db86b7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:05.693785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-419ce970-edac-449b-a1c7-b5592db86b7b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:05.695708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-419ce970-edac-449b-a1c7-b5592db86b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:05.773256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 478/959] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:05 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:05.826414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec43a222-dfcb-4f1a-befe-c8055a60b64e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:05.936974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec43a222-dfcb-4f1a-befe-c8055a60b64e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:05.975426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a58bb5-6b50-48b7-8377-2ccc6bae4949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:05.976104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a58bb5-6b50-48b7-8377-2ccc6bae4949 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:05.976468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a58bb5-6b50-48b7-8377-2ccc6bae4949 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:05.976690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a58bb5-6b50-48b7-8377-2ccc6bae4949 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:05.977330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a58bb5-6b50-48b7-8377-2ccc6bae4949 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:05.978762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a58bb5-6b50-48b7-8377-2ccc6bae4949 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-65a58bb5-6b50-48b7-8377-2ccc6bae4949 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:05.988166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:05.989330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:05.990181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 482/960] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:24:05 2022] GET /identity/v3/services/compute => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:05.996035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419ce970-edac-449b-a1c7-b5592db86b7b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:05.999668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:06.055048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a58bb5-6b50-48b7-8377-2ccc6bae4949 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:06.093909 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:06.105964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419ce970-edac-449b-a1c7-b5592db86b7b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:06.143709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23bbf85-29a0-45f8-8e9e-2f39bfb40624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:06.144611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23bbf85-29a0-45f8-8e9e-2f39bfb40624 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:06.144996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23bbf85-29a0-45f8-8e9e-2f39bfb40624 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:06.145369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23bbf85-29a0-45f8-8e9e-2f39bfb40624 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:06.146486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23bbf85-29a0-45f8-8e9e-2f39bfb40624 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:06.146610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23bbf85-29a0-45f8-8e9e-2f39bfb40624 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:06.148144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23bbf85-29a0-45f8-8e9e-2f39bfb40624 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:06.158082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 479/961] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:24:05 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:06.159423 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23bbf85-29a0-45f8-8e9e-2f39bfb40624 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:06.182709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a58bb5-6b50-48b7-8377-2ccc6bae4949 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:06.223931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19403cb8-7b34-4574-8f8f-031471518dc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:06.224898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19403cb8-7b34-4574-8f8f-031471518dc0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:06.227160 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19403cb8-7b34-4574-8f8f-031471518dc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:06.227603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19403cb8-7b34-4574-8f8f-031471518dc0 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:06.240688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19403cb8-7b34-4574-8f8f-031471518dc0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-19403cb8-7b34-4574-8f8f-031471518dc0 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:06.247783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:06.248977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:24:06.249864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:06.252699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:06.253771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 483/962] 10.222.0.25 () {56 vars in 1120 bytes} [Mon Sep 26 01:24:06 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:06.258949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19403cb8-7b34-4574-8f8f-031471518dc0 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:06.302429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23bbf85-29a0-45f8-8e9e-2f39bfb40624 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:06.356799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5055f34f-24a7-4ac7-bebc-27f99efc223d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:06.358108 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5055f34f-24a7-4ac7-bebc-27f99efc223d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:06.358608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5055f34f-24a7-4ac7-bebc-27f99efc223d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:06.359307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5055f34f-24a7-4ac7-bebc-27f99efc223d None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:06.360693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5055f34f-24a7-4ac7-bebc-27f99efc223d None admin] RBAC: Adding query filter params (type=compute) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:06.361017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5055f34f-24a7-4ac7-bebc-27f99efc223d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:06.362462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5055f34f-24a7-4ac7-bebc-27f99efc223d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:06.369684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:06.376510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 480/963] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:24:06 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:06.385896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19403cb8-7b34-4574-8f8f-031471518dc0 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:06.425319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313da9c4-26b1-4183-89e9-4293114d756b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:06.425941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313da9c4-26b1-4183-89e9-4293114d756b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:06.426225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313da9c4-26b1-4183-89e9-4293114d756b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:06.426501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313da9c4-26b1-4183-89e9-4293114d756b None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:06.427183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313da9c4-26b1-4183-89e9-4293114d756b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:06.427489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313da9c4-26b1-4183-89e9-4293114d756b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:06.428926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313da9c4-26b1-4183-89e9-4293114d756b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:06.439622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 484/964] 10.222.0.25 () {56 vars in 1115 bytes} [Mon Sep 26 01:24:06 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:06.445731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5055f34f-24a7-4ac7-bebc-27f99efc223d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:06.467676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc9465a-6bed-4bd6-a5ce-0aaea4332441 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:06.468010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc9465a-6bed-4bd6-a5ce-0aaea4332441 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:06.468235 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc9465a-6bed-4bd6-a5ce-0aaea4332441 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:06.469912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 485/965] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:24:06.475594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23418c0a-0f85-4616-b21a-ed752542822f None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:06.475875 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23418c0a-0f85-4616-b21a-ed752542822f None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:06.476564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23418c0a-0f85-4616-b21a-ed752542822f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:06.500705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-23418c0a-0f85-4616-b21a-ed752542822f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:06.501942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-23418c0a-0f85-4616-b21a-ed752542822f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:06.590023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5055f34f-24a7-4ac7-bebc-27f99efc223d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:06.590862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 486/966] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:06 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:06.629637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:06.636205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e21a350-e911-4cea-b6bc-6c03e208082b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:06.636903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e21a350-e911-4cea-b6bc-6c03e208082b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:06.638014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e21a350-e911-4cea-b6bc-6c03e208082b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:06.638311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e21a350-e911-4cea-b6bc-6c03e208082b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:06.649308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e21a350-e911-4cea-b6bc-6c03e208082b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:06.650760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e21a350-e911-4cea-b6bc-6c03e208082b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-5e21a350-e911-4cea-b6bc-6c03e208082b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:06.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:06.666873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:06.668324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:06.668324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:06.668324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:06.668324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 481/967] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:06.673735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23418c0a-0f85-4616-b21a-ed752542822f None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:06.759306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:06.780208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23418c0a-0f85-4616-b21a-ed752542822f None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:06.816162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddd8031-8faf-41fe-b331-b7a2b8d521ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:06.817131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd8031-8faf-41fe-b331-b7a2b8d521ca None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:06.817601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd8031-8faf-41fe-b331-b7a2b8d521ca None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:06.818055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd8031-8faf-41fe-b331-b7a2b8d521ca None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:06.818893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddd8031-8faf-41fe-b331-b7a2b8d521ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:06.819386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddd8031-8faf-41fe-b331-b7a2b8d521ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:06.821537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddd8031-8faf-41fe-b331-b7a2b8d521ca None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:06.834174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 487/968] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:06.844376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e21a350-e911-4cea-b6bc-6c03e208082b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:06.873499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddd8031-8faf-41fe-b331-b7a2b8d521ca None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:06.954583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e21a350-e911-4cea-b6bc-6c03e208082b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:06.996620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea26eb3d-22eb-4c1c-afc4-8ecff5e58b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:06.997660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea26eb3d-22eb-4c1c-afc4-8ecff5e58b6f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:06.997943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea26eb3d-22eb-4c1c-afc4-8ecff5e58b6f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:06.998214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea26eb3d-22eb-4c1c-afc4-8ecff5e58b6f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:07.006416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddd8031-8faf-41fe-b331-b7a2b8d521ca None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:07.008676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea26eb3d-22eb-4c1c-afc4-8ecff5e58b6f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:07.011016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea26eb3d-22eb-4c1c-afc4-8ecff5e58b6f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ea26eb3d-22eb-4c1c-afc4-8ecff5e58b6f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:24:07.026039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:07.027042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:07.028418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 482/969] 10.222.0.25 () {56 vars in 1117 bytes} [Mon Sep 26 01:24:06 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:07.033013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea26eb3d-22eb-4c1c-afc4-8ecff5e58b6f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:07.060121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9018ce94-4c94-47fa-a6a9-9259ceedeb18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:07.060769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9018ce94-4c94-47fa-a6a9-9259ceedeb18 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:07.061052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9018ce94-4c94-47fa-a6a9-9259ceedeb18 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:07.061330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9018ce94-4c94-47fa-a6a9-9259ceedeb18 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:07.061917 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9018ce94-4c94-47fa-a6a9-9259ceedeb18 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:07.064167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9018ce94-4c94-47fa-a6a9-9259ceedeb18 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-9018ce94-4c94-47fa-a6a9-9259ceedeb18 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:07.088601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:07.089802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:07.090713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:07.096184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 488/970] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:24:06 2022] GET /identity/v3/services/object-store => generated 93 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:07.097785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9018ce94-4c94-47fa-a6a9-9259ceedeb18 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:07.112139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:07.124080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:07.199449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea26eb3d-22eb-4c1c-afc4-8ecff5e58b6f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:07.246057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a063e0-4fcf-4c97-a65c-ed1add877a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:07.246902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a063e0-4fcf-4c97-a65c-ed1add877a0e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:07.247333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a063e0-4fcf-4c97-a65c-ed1add877a0e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:07.247752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a063e0-4fcf-4c97-a65c-ed1add877a0e None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:07.248660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a063e0-4fcf-4c97-a65c-ed1add877a0e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:07.249135 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a063e0-4fcf-4c97-a65c-ed1add877a0e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:07.251228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a063e0-4fcf-4c97-a65c-ed1add877a0e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:07.272758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 483/971] 10.222.0.25 () {56 vars in 1149 bytes} [Mon Sep 26 01:24:07 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:07.281697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a063e0-4fcf-4c97-a65c-ed1add877a0e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:07.288972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9018ce94-4c94-47fa-a6a9-9259ceedeb18 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:07.365448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079f9514-6f3c-4d79-9186-8435bf76fa27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:07.366125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079f9514-6f3c-4d79-9186-8435bf76fa27 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:07.366407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079f9514-6f3c-4d79-9186-8435bf76fa27 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:07.367310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079f9514-6f3c-4d79-9186-8435bf76fa27 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:07.368029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079f9514-6f3c-4d79-9186-8435bf76fa27 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:07.368335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079f9514-6f3c-4d79-9186-8435bf76fa27 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:07.374164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079f9514-6f3c-4d79-9186-8435bf76fa27 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:07.384398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 489/972] 10.222.0.25 () {56 vars in 1122 bytes} [Mon Sep 26 01:24:07 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:07.388329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079f9514-6f3c-4d79-9186-8435bf76fa27 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:07.465069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a063e0-4fcf-4c97-a65c-ed1add877a0e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:07.466796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:07.513149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:07.548806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62ea8de-4ef7-4f6f-8fe4-be87840bad24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:07.550261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62ea8de-4ef7-4f6f-8fe4-be87840bad24 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:07.550710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62ea8de-4ef7-4f6f-8fe4-be87840bad24 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:07.551156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62ea8de-4ef7-4f6f-8fe4-be87840bad24 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:07.552465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079f9514-6f3c-4d79-9186-8435bf76fa27 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:07.599473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80082083-90dd-413c-b508-9c37966d396d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:07.600223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80082083-90dd-413c-b508-9c37966d396d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:07.600623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80082083-90dd-413c-b508-9c37966d396d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:07.602438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80082083-90dd-413c-b508-9c37966d396d None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:07.603182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80082083-90dd-413c-b508-9c37966d396d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:07.603627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80082083-90dd-413c-b508-9c37966d396d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:07.612056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80082083-90dd-413c-b508-9c37966d396d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:07.629298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 490/973] 10.222.0.25 () {56 vars in 1122 bytes} [Mon Sep 26 01:24:07 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:07.631125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80082083-90dd-413c-b508-9c37966d396d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:07.712914 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62ea8de-4ef7-4f6f-8fe4-be87840bad24 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:07.715402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62ea8de-4ef7-4f6f-8fe4-be87840bad24 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:07.752675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80082083-90dd-413c-b508-9c37966d396d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:07.787566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:07.795410 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11eab1ab-58ed-4bdf-a4b0-df91a95fa412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:07.796319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eab1ab-58ed-4bdf-a4b0-df91a95fa412 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:07.796967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eab1ab-58ed-4bdf-a4b0-df91a95fa412 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:07.797444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eab1ab-58ed-4bdf-a4b0-df91a95fa412 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:07.798903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11eab1ab-58ed-4bdf-a4b0-df91a95fa412 None admin] RBAC: Adding query filter params (interface=admin, service_id=a46c30fe636844b7be5e8d1382b68d48, region_id=RegionOne) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:07.799433 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11eab1ab-58ed-4bdf-a4b0-df91a95fa412 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:07.801029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11eab1ab-58ed-4bdf-a4b0-df91a95fa412 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:07.812025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 491/974] 10.222.0.25 () {56 vars in 1249 bytes} [Mon Sep 26 01:24:07 2022] GET /identity/v3/endpoints?service_id=a46c30fe636844b7be5e8d1382b68d48&interface=admin®ion_id=RegionOne => generated 193 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:07.817811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11eab1ab-58ed-4bdf-a4b0-df91a95fa412 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:07.853354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 484/975] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:07 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 572 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:07.859253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62ea8de-4ef7-4f6f-8fe4-be87840bad24 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:07.878659 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:07.941301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11eab1ab-58ed-4bdf-a4b0-df91a95fa412 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:07.969766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62ea8de-4ef7-4f6f-8fe4-be87840bad24 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:07.979189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2be043-ed17-465a-a992-901939b0af5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:07.979799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2be043-ed17-465a-a992-901939b0af5c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:07.980081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2be043-ed17-465a-a992-901939b0af5c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:07.980371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2be043-ed17-465a-a992-901939b0af5c None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:07.981171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2be043-ed17-465a-a992-901939b0af5c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:07.982534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2be043-ed17-465a-a992-901939b0af5c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:07.988606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 492/976] 10.222.0.25 () {56 vars in 1087 bytes} [Mon Sep 26 01:24:07 2022] GET /identity/v3/services => generated 1497 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:24:07.989604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1f66eb-4b18-4859-9606-0c76f0e78f60 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:07.989892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1f66eb-4b18-4859-9606-0c76f0e78f60 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:07.990169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1f66eb-4b18-4859-9606-0c76f0e78f60 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:07.991501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 493/977] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:07 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:07.997260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac6d5b5-757e-481f-88b8-9d126f8311d2 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:07.997595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac6d5b5-757e-481f-88b8-9d126f8311d2 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:07.997815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac6d5b5-757e-481f-88b8-9d126f8311d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.017618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11466fc-e233-45d0-9523-e0fde2680b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:08.018848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-9ac6d5b5-757e-481f-88b8-9d126f8311d2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:08.019290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11466fc-e233-45d0-9523-e0fde2680b50 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:08.019805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11466fc-e233-45d0-9523-e0fde2680b50 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:08.020097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9ac6d5b5-757e-481f-88b8-9d126f8311d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:08.020554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11466fc-e233-45d0-9523-e0fde2680b50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.041691 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11466fc-e233-45d0-9523-e0fde2680b50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:08.043885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11466fc-e233-45d0-9523-e0fde2680b50 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f11466fc-e233-45d0-9523-e0fde2680b50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:08.069155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:08.070318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:08.071351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:08.071351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:08.071351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:08.078830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 485/978] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:08.079888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11466fc-e233-45d0-9523-e0fde2680b50 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:08.113965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:08.118402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 494/979] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:07 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:24:08.152852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:08.208792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11466fc-e233-45d0-9523-e0fde2680b50 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:08.244507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd4aa2-ac66-4a41-a930-d0cc25ad0e9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:08.245391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd4aa2-ac66-4a41-a930-d0cc25ad0e9e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:08.245842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd4aa2-ac66-4a41-a930-d0cc25ad0e9e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:08.246274 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd4aa2-ac66-4a41-a930-d0cc25ad0e9e None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.247072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd4aa2-ac66-4a41-a930-d0cc25ad0e9e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:08.247522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd4aa2-ac66-4a41-a930-d0cc25ad0e9e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:08.249076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd4aa2-ac66-4a41-a930-d0cc25ad0e9e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:08.261907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 486/980] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:08.267673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac6d5b5-757e-481f-88b8-9d126f8311d2 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:08.372990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac6d5b5-757e-481f-88b8-9d126f8311d2 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:08.402016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd4aa2-ac66-4a41-a930-d0cc25ad0e9e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:08.412783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cae0a-7fb6-4844-a5f6-dfca4e359d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:08.413957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cae0a-7fb6-4844-a5f6-dfca4e359d77 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:08.414250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cae0a-7fb6-4844-a5f6-dfca4e359d77 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:08.414696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cae0a-7fb6-4844-a5f6-dfca4e359d77 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.422521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8cae0a-7fb6-4844-a5f6-dfca4e359d77 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:08.424089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8cae0a-7fb6-4844-a5f6-dfca4e359d77 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-bc8cae0a-7fb6-4844-a5f6-dfca4e359d77 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:24:08.437440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:08.438404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:08.439404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:24:08.439404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:24:08.439404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:24:08.439404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:24:08.439404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:24:08.439404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:24:08.439404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:08.439404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:08.439876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 495/981] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:24:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:08.446981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cae0a-7fb6-4844-a5f6-dfca4e359d77 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:08.503585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:08.543563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd4aa2-ac66-4a41-a930-d0cc25ad0e9e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:08.559941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:08.572461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cae0a-7fb6-4844-a5f6-dfca4e359d77 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:08.584054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e5703d-3dcd-4ae4-b7af-f070ee688a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:08.584697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e5703d-3dcd-4ae4-b7af-f070ee688a87 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:08.584973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e5703d-3dcd-4ae4-b7af-f070ee688a87 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:08.585239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e5703d-3dcd-4ae4-b7af-f070ee688a87 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.586135 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e5703d-3dcd-4ae4-b7af-f070ee688a87 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:08.587568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e5703d-3dcd-4ae4-b7af-f070ee688a87 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:08.602713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.sql.core [None req-b5e5703d-3dcd-4ae4-b7af-f070ee688a87 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 26 01:24:08.602713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 26 01:24:08.602713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [parameters: {'user_id': '4d56dc90cca04b08b5767788a5c5c2e3', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 26 01:24:08.602713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81423) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b5e5703d-3dcd-4ae4-b7af-f070ee688a87 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result = self._query(query) Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application conn.query(q) Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result.read() Sep 26 01:24:08.609637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 26 01:24:08.610572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application next(self.gen) Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application yield resource Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application next(self.gen) Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.session.rollback() Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.force_reraise() Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise self.value Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application session.commit() Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 26 01:24:08.611604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._prepare_impl() Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.session.flush() Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._flush(objects) Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application compat.raise_( Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application flush_context.execute() Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rec.execute(self) Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 26 01:24:08.612570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 26 01:24:08.613530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result = self._query(query) Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application conn.query(q) Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application result.read() Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 26 01:24:08.614425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [parameters: {'user_id': '4d56dc90cca04b08b5767788a5c5c2e3', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:08.615284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 26 01:24:08.616121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 26 01:24:08.620473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 26 01:24:08.620473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 26 01:24:08.620473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 26 01:24:08.620473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 26 01:24:08.620473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 26 01:24:08.620473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:08.620473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 487/982] 10.222.0.25 () {60 vars in 1131 bytes} [Mon Sep 26 01:24:08 2022] POST /identity/v3/users => generated 160 bytes in 209 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:08.620473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e5703d-3dcd-4ae4-b7af-f070ee688a87 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:08.622685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6dbbd8-009c-491c-951d-a7c180ae8b26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:08.623378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6dbbd8-009c-491c-951d-a7c180ae8b26 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:08.623720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6dbbd8-009c-491c-951d-a7c180ae8b26 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:08.626347 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6dbbd8-009c-491c-951d-a7c180ae8b26 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.629796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6dbbd8-009c-491c-951d-a7c180ae8b26 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:08.629796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6dbbd8-009c-491c-951d-a7c180ae8b26 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:08.629796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6dbbd8-009c-491c-951d-a7c180ae8b26 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:08.651473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 496/983] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:24:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:08.654622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6dbbd8-009c-491c-951d-a7c180ae8b26 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:08.743413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e5703d-3dcd-4ae4-b7af-f070ee688a87 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:08.767789 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6dbbd8-009c-491c-951d-a7c180ae8b26 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:08.779479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:08.784449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902947b0-632e-4184-8365-fdc819e2d190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:08.785243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902947b0-632e-4184-8365-fdc819e2d190 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:08.785530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902947b0-632e-4184-8365-fdc819e2d190 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:08.785823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902947b0-632e-4184-8365-fdc819e2d190 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.793474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902947b0-632e-4184-8365-fdc819e2d190 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:08.795135 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902947b0-632e-4184-8365-fdc819e2d190 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-902947b0-632e-4184-8365-fdc819e2d190 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:24:08.807792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:08.808982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:08.810185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:08.810185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:08.810185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:24:08.810185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:24:08.810185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:24:08.810185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:24:08.810185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:24:08.810185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:24:08.810185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 26 01:24:08.810185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:08.810872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558a2b5a-417c-4cde-a2bf-732c5046e3e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:08.811651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a2b5a-417c-4cde-a2bf-732c5046e3e8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:08.812028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a2b5a-417c-4cde-a2bf-732c5046e3e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:08.812447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a2b5a-417c-4cde-a2bf-732c5046e3e8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.813280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558a2b5a-417c-4cde-a2bf-732c5046e3e8 None admin] RBAC: Adding query filter params (role.id=fe72ae0ebcab44b69a769caf591d6eda, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=50a8d0782b6f47d1aec027c6e15018a8) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:08.813732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558a2b5a-417c-4cde-a2bf-732c5046e3e8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:08.813831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 488/984] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:24:08 2022] GET /identity/v3/users/cinder => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:08.815114 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd177e83-c5eb-4fff-9e43-63e0fab94eff None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:08.815403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd177e83-c5eb-4fff-9e43-63e0fab94eff None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:08.815533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558a2b5a-417c-4cde-a2bf-732c5046e3e8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:08.815683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd177e83-c5eb-4fff-9e43-63e0fab94eff None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.820097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 489/985] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:08 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:08.820847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd177e83-c5eb-4fff-9e43-63e0fab94eff None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:08.826345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 497/986] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:24:08 2022] GET /identity/v3/role_assignments?user.id=50a8d0782b6f47d1aec027c6e15018a8&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=fe72ae0ebcab44b69a769caf591d6eda => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:08.827604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2f9caa-5ed6-49a6-b24c-5783eddb62ff None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:08.827893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2f9caa-5ed6-49a6-b24c-5783eddb62ff None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:08.828169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2f9caa-5ed6-49a6-b24c-5783eddb62ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.852282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-9f2f9caa-5ed6-49a6-b24c-5783eddb62ff None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:08.854954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9f2f9caa-5ed6-49a6-b24c-5783eddb62ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:08.867951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:08.946407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 498/987] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:08 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:08.952583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd177e83-c5eb-4fff-9e43-63e0fab94eff None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:08.993369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62af133a-dff3-438d-ab8f-833a7dcac5f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:08.994368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62af133a-dff3-438d-ab8f-833a7dcac5f2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:08.994879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62af133a-dff3-438d-ab8f-833a7dcac5f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:08.995458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62af133a-dff3-438d-ab8f-833a7dcac5f2 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:08.996439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62af133a-dff3-438d-ab8f-833a7dcac5f2 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:08.996981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62af133a-dff3-438d-ab8f-833a7dcac5f2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:08.998751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62af133a-dff3-438d-ab8f-833a7dcac5f2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:09.042444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 490/988] 10.222.0.25 () {56 vars in 1101 bytes} [Mon Sep 26 01:24:08 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:09.221438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2f9caa-5ed6-49a6-b24c-5783eddb62ff None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:09.324334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2f9caa-5ed6-49a6-b24c-5783eddb62ff None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:09.326808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:09.361363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd674c0-9262-4a55-9011-baabde5f091a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:09.361971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd674c0-9262-4a55-9011-baabde5f091a None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:09.362253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd674c0-9262-4a55-9011-baabde5f091a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:09.362532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd674c0-9262-4a55-9011-baabde5f091a None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:09.363196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd674c0-9262-4a55-9011-baabde5f091a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:09.364597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd674c0-9262-4a55-9011-baabde5f091a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:09.379893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 499/989] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:24:09 2022] POST /identity/v3/services => generated 242 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:24:10.308105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeee1560-1ad8-45e7-aa81-d7c93703d6b6 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:10.308669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeee1560-1ad8-45e7-aa81-d7c93703d6b6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:10.308669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeee1560-1ad8-45e7-aa81-d7c93703d6b6 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:10.310115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 491/990] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:10.315128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3531a78b-5789-460f-b51f-08d743c2acff None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:10.315418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3531a78b-5789-460f-b51f-08d743c2acff None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:10.315696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3531a78b-5789-460f-b51f-08d743c2acff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:10.333884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-3531a78b-5789-460f-b51f-08d743c2acff None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:10.335054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3531a78b-5789-460f-b51f-08d743c2acff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:10.410758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 500/991] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:10 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:10.682597 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeee1560-1ad8-45e7-aa81-d7c93703d6b6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:10.781550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:10.790693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeee1560-1ad8-45e7-aa81-d7c93703d6b6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:10.825164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706c67e-547e-4053-befc-315ad6637c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:10.825793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706c67e-547e-4053-befc-315ad6637c59 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:10.826069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706c67e-547e-4053-befc-315ad6637c59 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:10.826337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706c67e-547e-4053-befc-315ad6637c59 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:10.826911 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b706c67e-547e-4053-befc-315ad6637c59 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:10.828302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b706c67e-547e-4053-befc-315ad6637c59 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b706c67e-547e-4053-befc-315ad6637c59 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:10.837544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:10.838579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:10.839447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 492/992] 10.222.0.25 () {56 vars in 1126 bytes} [Mon Sep 26 01:24:10 2022] GET /identity/v3/services/object-store => generated 93 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:10.846206 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3531a78b-5789-460f-b51f-08d743c2acff None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:10.938836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:10.952081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3531a78b-5789-460f-b51f-08d743c2acff None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:10.990675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e505a3-4dae-4451-bfac-51661e3524d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:10.991285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e505a3-4dae-4451-bfac-51661e3524d1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:10.992201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e505a3-4dae-4451-bfac-51661e3524d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:10.992569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e505a3-4dae-4451-bfac-51661e3524d1 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:10.993211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e505a3-4dae-4451-bfac-51661e3524d1 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:10.993517 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e505a3-4dae-4451-bfac-51661e3524d1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:10.994885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e505a3-4dae-4451-bfac-51661e3524d1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:11.004319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 501/993] 10.222.0.25 () {56 vars in 1122 bytes} [Mon Sep 26 01:24:10 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:11.005864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706c67e-547e-4053-befc-315ad6637c59 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:11.072633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b79622-2af6-411c-a3fd-e6745bcc7df1 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:11.072960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b79622-2af6-411c-a3fd-e6745bcc7df1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:11.073195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b79622-2af6-411c-a3fd-e6745bcc7df1 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:11.074669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 502/994] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:24:11.079365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29967147-937f-4381-8801-c095ab34ef2e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:11.079675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29967147-937f-4381-8801-c095ab34ef2e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:11.079948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29967147-937f-4381-8801-c095ab34ef2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:11.098360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-29967147-937f-4381-8801-c095ab34ef2e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:11.099563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-29967147-937f-4381-8801-c095ab34ef2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:11.121346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706c67e-547e-4053-befc-315ad6637c59 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:11.159096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cb5321-d0af-42a2-b95e-ed3219002724 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:11.159730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cb5321-d0af-42a2-b95e-ed3219002724 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:11.160008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cb5321-d0af-42a2-b95e-ed3219002724 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:11.160280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cb5321-d0af-42a2-b95e-ed3219002724 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:11.160946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cb5321-d0af-42a2-b95e-ed3219002724 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:11.161248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cb5321-d0af-42a2-b95e-ed3219002724 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:11.162646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cb5321-d0af-42a2-b95e-ed3219002724 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:11.168086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 493/995] 10.222.0.25 () {56 vars in 1122 bytes} [Mon Sep 26 01:24:11 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:11.172391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 503/996] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:11 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:24:11.173816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cb5321-d0af-42a2-b95e-ed3219002724 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:11.200982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:11.278408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cb5321-d0af-42a2-b95e-ed3219002724 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:11.312952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8290b3-9184-4eae-980e-0df1c522768a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:11.313564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8290b3-9184-4eae-980e-0df1c522768a None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:11.313819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8290b3-9184-4eae-980e-0df1c522768a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:11.314092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8290b3-9184-4eae-980e-0df1c522768a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:11.314765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8290b3-9184-4eae-980e-0df1c522768a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:11.316127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8290b3-9184-4eae-980e-0df1c522768a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:11.334232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 494/997] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:24:11 2022] POST /identity/v3/endpoints => generated 325 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:24:11.440297 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29967147-937f-4381-8801-c095ab34ef2e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:11.534081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:11.552073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c8dc7-97d6-4b53-83b2-525b90c35231 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:11.552396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c8dc7-97d6-4b53-83b2-525b90c35231 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:11.552658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c8dc7-97d6-4b53-83b2-525b90c35231 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:11.553083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29967147-937f-4381-8801-c095ab34ef2e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:11.554052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 495/998] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:24:11.560448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e355c-5344-4340-83c8-0b018315f438 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:11.560723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e355c-5344-4340-83c8-0b018315f438 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:11.560997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e355c-5344-4340-83c8-0b018315f438 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:11.581993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d21e355c-5344-4340-83c8-0b018315f438 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:11.583830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d21e355c-5344-4340-83c8-0b018315f438 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:11.596982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d98a23-a56d-4ae7-a519-01d8251f4074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:11.597621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d98a23-a56d-4ae7-a519-01d8251f4074 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:11.597903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d98a23-a56d-4ae7-a519-01d8251f4074 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:11.598180 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d98a23-a56d-4ae7-a519-01d8251f4074 None admin] PATH_INFO: `/v3/services/image` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:11.598766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d98a23-a56d-4ae7-a519-01d8251f4074 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:11.600129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d98a23-a56d-4ae7-a519-01d8251f4074 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-20d98a23-a56d-4ae7-a519-01d8251f4074 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:11.609546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:11.610764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:11.611757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 504/999] 10.222.0.25 () {56 vars in 1105 bytes} [Mon Sep 26 01:24:11 2022] GET /identity/v3/services/image => generated 86 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:11.616438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d98a23-a56d-4ae7-a519-01d8251f4074 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:11.660198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 496/1000] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:11 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:11.727342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d98a23-a56d-4ae7-a519-01d8251f4074 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:11.762306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebc8ffe-74f9-431d-9a6b-292d0d7570ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:11.762910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebc8ffe-74f9-431d-9a6b-292d0d7570ef None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:11.763189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebc8ffe-74f9-431d-9a6b-292d0d7570ef None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:11.763461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebc8ffe-74f9-431d-9a6b-292d0d7570ef None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:11.764082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebc8ffe-74f9-431d-9a6b-292d0d7570ef None admin] RBAC: Adding query filter params (name=image) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:11.764412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebc8ffe-74f9-431d-9a6b-292d0d7570ef None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:11.765772 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebc8ffe-74f9-431d-9a6b-292d0d7570ef None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:11.770596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 505/1001] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:11 2022] GET /identity/v3/services?name=image => generated 122 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:11.776310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21e355c-5344-4340-83c8-0b018315f438 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:11.790993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:11.884711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21e355c-5344-4340-83c8-0b018315f438 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:11.919389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd489226-ca9c-484a-bfec-6996a6e82998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:11.920045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd489226-ca9c-484a-bfec-6996a6e82998 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:11.920332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd489226-ca9c-484a-bfec-6996a6e82998 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:11.920632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd489226-ca9c-484a-bfec-6996a6e82998 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:11.921261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd489226-ca9c-484a-bfec-6996a6e82998 None admin] RBAC: Adding query filter params (type=image) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:11.921567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd489226-ca9c-484a-bfec-6996a6e82998 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:11.922943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd489226-ca9c-484a-bfec-6996a6e82998 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:11.932441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 497/1002] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:11 2022] GET /identity/v3/services?type=image => generated 122 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:11.938488 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebc8ffe-74f9-431d-9a6b-292d0d7570ef None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:11.952545 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:12.057100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebc8ffe-74f9-431d-9a6b-292d0d7570ef None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:12.095895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bb18e8-80a6-441d-b2c6-e4e434b98339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:12.096560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bb18e8-80a6-441d-b2c6-e4e434b98339 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:12.096844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bb18e8-80a6-441d-b2c6-e4e434b98339 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:12.097119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23bb18e8-80a6-441d-b2c6-e4e434b98339 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:12.097709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bb18e8-80a6-441d-b2c6-e4e434b98339 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:12.099080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23bb18e8-80a6-441d-b2c6-e4e434b98339 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-23bb18e8-80a6-441d-b2c6-e4e434b98339 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:12.109117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:12.110211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:12.111596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 506/1003] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:24:11 2022] GET /identity/v3/services/compute => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:12.116180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd489226-ca9c-484a-bfec-6996a6e82998 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:12.198843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:12.224021 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd489226-ca9c-484a-bfec-6996a6e82998 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:12.261058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ef274-5d70-4c00-8886-4618c390c53b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:12.261671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ef274-5d70-4c00-8886-4618c390c53b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:12.261953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ef274-5d70-4c00-8886-4618c390c53b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:12.262252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39ef274-5d70-4c00-8886-4618c390c53b None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:12.262885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ef274-5d70-4c00-8886-4618c390c53b None admin] RBAC: Adding query filter params (name=compute) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:12.263202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ef274-5d70-4c00-8886-4618c390c53b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:12.264601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39ef274-5d70-4c00-8886-4618c390c53b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:12.275099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 498/1004] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:24:12 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:12.276541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bb18e8-80a6-441d-b2c6-e4e434b98339 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:12.348075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:12.378547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23bb18e8-80a6-441d-b2c6-e4e434b98339 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:12.414422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fcd903-516d-4f41-ac0a-8982a0443442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:12.415054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fcd903-516d-4f41-ac0a-8982a0443442 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:12.415336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fcd903-516d-4f41-ac0a-8982a0443442 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:12.415611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fcd903-516d-4f41-ac0a-8982a0443442 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:12.416246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fcd903-516d-4f41-ac0a-8982a0443442 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:12.416600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fcd903-516d-4f41-ac0a-8982a0443442 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:12.417971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fcd903-516d-4f41-ac0a-8982a0443442 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:12.425020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 507/1005] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:24:12 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:12.429866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ef274-5d70-4c00-8886-4618c390c53b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:12.537583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39ef274-5d70-4c00-8886-4618c390c53b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:12.573469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98298aba-38bd-4671-a6ad-d5e9b585d684 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:12.574071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98298aba-38bd-4671-a6ad-d5e9b585d684 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:12.574345 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98298aba-38bd-4671-a6ad-d5e9b585d684 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:12.574611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98298aba-38bd-4671-a6ad-d5e9b585d684 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:12.575305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98298aba-38bd-4671-a6ad-d5e9b585d684 None admin] RBAC: Adding query filter params (interface=public, service_id=427d161cff8a407b96bbca86d777ba7c, region_id=RegionOne) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:12.575608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98298aba-38bd-4671-a6ad-d5e9b585d684 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:12.576994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98298aba-38bd-4671-a6ad-d5e9b585d684 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:12.587909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 499/1006] 10.222.0.25 () {56 vars in 1251 bytes} [Mon Sep 26 01:24:12 2022] GET /identity/v3/endpoints?service_id=427d161cff8a407b96bbca86d777ba7c&interface=public®ion_id=RegionOne => generated 194 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:12.589135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fcd903-516d-4f41-ac0a-8982a0443442 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:12.696811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fcd903-516d-4f41-ac0a-8982a0443442 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:12.731323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f958b67-ef95-4eaa-92b3-d8c8dba19486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:12.731928 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f958b67-ef95-4eaa-92b3-d8c8dba19486 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:12.732204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f958b67-ef95-4eaa-92b3-d8c8dba19486 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:12.732502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f958b67-ef95-4eaa-92b3-d8c8dba19486 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:12.733099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f958b67-ef95-4eaa-92b3-d8c8dba19486 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:12.734448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f958b67-ef95-4eaa-92b3-d8c8dba19486 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:12.736215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:12.748377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 508/1007] 10.222.0.25 () {56 vars in 1087 bytes} [Mon Sep 26 01:24:12 2022] GET /identity/v3/services => generated 1727 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:24:13.530155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c8a79-abac-4e91-9e68-8705ac4fd67d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:13.530718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c8a79-abac-4e91-9e68-8705ac4fd67d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:13.530718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c8a79-abac-4e91-9e68-8705ac4fd67d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:13.532034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 500/1008] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:13.537570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66bb231-332b-4f21-8eed-5aff21deb082 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:13.537946 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66bb231-332b-4f21-8eed-5aff21deb082 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:13.538829 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66bb231-332b-4f21-8eed-5aff21deb082 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:13.557665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b66bb231-332b-4f21-8eed-5aff21deb082 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:13.558859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b66bb231-332b-4f21-8eed-5aff21deb082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:13.629256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 509/1009] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:13 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:13.897114 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c8a79-abac-4e91-9e68-8705ac4fd67d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:13.901085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:14.002911 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c8a79-abac-4e91-9e68-8705ac4fd67d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:14.050441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c14561-3cd9-4f1e-87ea-afeff6dea4b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:14.051084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c14561-3cd9-4f1e-87ea-afeff6dea4b2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:14.051363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c14561-3cd9-4f1e-87ea-afeff6dea4b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:14.051634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c14561-3cd9-4f1e-87ea-afeff6dea4b2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:14.052785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c14561-3cd9-4f1e-87ea-afeff6dea4b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:14.054775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c14561-3cd9-4f1e-87ea-afeff6dea4b2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:14.058434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 501/1010] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:14.062475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66bb231-332b-4f21-8eed-5aff21deb082 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:14.126971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4e4a7-36e8-4cbd-b53d-92a6b109ec3f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:14.127485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4e4a7-36e8-4cbd-b53d-92a6b109ec3f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:14.127964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4e4a7-36e8-4cbd-b53d-92a6b109ec3f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:14.129541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 502/1011] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:24:14.134771 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed22584c-75cd-4d22-a156-1b5368f5f7db None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:14.135052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed22584c-75cd-4d22-a156-1b5368f5f7db None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:14.135489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed22584c-75cd-4d22-a156-1b5368f5f7db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:14.154043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ed22584c-75cd-4d22-a156-1b5368f5f7db None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:14.155163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ed22584c-75cd-4d22-a156-1b5368f5f7db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:14.168577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66bb231-332b-4f21-8eed-5aff21deb082 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:14.200238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:14.207434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13845375-ae36-49ea-9f5f-ea4fce6deaff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:14.208051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13845375-ae36-49ea-9f5f-ea4fce6deaff None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:14.208329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13845375-ae36-49ea-9f5f-ea4fce6deaff None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:14.208631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13845375-ae36-49ea-9f5f-ea4fce6deaff None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:14.209273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13845375-ae36-49ea-9f5f-ea4fce6deaff None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:14.211796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13845375-ae36-49ea-9f5f-ea4fce6deaff None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:14.227438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 503/1012] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:14 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:14.253799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 510/1013] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:24:14 2022] POST /identity/v3/projects => generated 311 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:24:14.502004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed22584c-75cd-4d22-a156-1b5368f5f7db None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:14.504187 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:14.607734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed22584c-75cd-4d22-a156-1b5368f5f7db None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:14.648767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e30d230-61bb-4676-98e8-fce390c60f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:14.649624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e30d230-61bb-4676-98e8-fce390c60f4a None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:14.650055 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e30d230-61bb-4676-98e8-fce390c60f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:14.650474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e30d230-61bb-4676-98e8-fce390c60f4a None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:14.651320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e30d230-61bb-4676-98e8-fce390c60f4a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:14.652973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e30d230-61bb-4676-98e8-fce390c60f4a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:14.670421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 504/1014] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:24:14 2022] POST /identity/v3/services => generated 242 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:24:14.932249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d66260c-83f5-41d1-9be4-bc187ea176c2 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:14.932773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d66260c-83f5-41d1-9be4-bc187ea176c2 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:14.933102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d66260c-83f5-41d1-9be4-bc187ea176c2 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:14.935204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 511/1015] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:14.940982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17941550-426f-47a2-91e2-4db25f12bf07 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:14.941265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17941550-426f-47a2-91e2-4db25f12bf07 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:14.941538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17941550-426f-47a2-91e2-4db25f12bf07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:14.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-17941550-426f-47a2-91e2-4db25f12bf07 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:14.963721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-17941550-426f-47a2-91e2-4db25f12bf07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:15.031652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:15.065603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 505/1016] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:14 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:15.374398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d66260c-83f5-41d1-9be4-bc187ea176c2 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:15.489834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d66260c-83f5-41d1-9be4-bc187ea176c2 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:15.525358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f09aa2a-34c6-43a9-a6ab-1f5ea6d87942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:15.526015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f09aa2a-34c6-43a9-a6ab-1f5ea6d87942 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:15.526316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f09aa2a-34c6-43a9-a6ab-1f5ea6d87942 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:15.526613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f09aa2a-34c6-43a9-a6ab-1f5ea6d87942 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:15.527187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f09aa2a-34c6-43a9-a6ab-1f5ea6d87942 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:15.528572 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f09aa2a-34c6-43a9-a6ab-1f5ea6d87942 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-7f09aa2a-34c6-43a9-a6ab-1f5ea6d87942 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:15.537337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:15.538460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 26 01:24:15.539424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:15.540915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 512/1017] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:24:15 2022] GET /identity/v3/services/compute => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:15.544406 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17941550-426f-47a2-91e2-4db25f12bf07 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:15.645843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17941550-426f-47a2-91e2-4db25f12bf07 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:15.685661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcec706-2b16-4426-8052-fee7a6e505b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:15.686289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcec706-2b16-4426-8052-fee7a6e505b1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:15.686566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcec706-2b16-4426-8052-fee7a6e505b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:15.686839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcec706-2b16-4426-8052-fee7a6e505b1 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:15.687488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fcec706-2b16-4426-8052-fee7a6e505b1 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:15.687792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fcec706-2b16-4426-8052-fee7a6e505b1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:15.689269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fcec706-2b16-4426-8052-fee7a6e505b1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:15.704146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 506/1018] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:24:15 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:15.705466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f09aa2a-34c6-43a9-a6ab-1f5ea6d87942 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:15.755265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:15.813311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f09aa2a-34c6-43a9-a6ab-1f5ea6d87942 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:15.846515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f72ec8-e47d-49da-a0d3-1b3ec58ef26c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:15.847118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f72ec8-e47d-49da-a0d3-1b3ec58ef26c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:15.847400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f72ec8-e47d-49da-a0d3-1b3ec58ef26c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:15.847676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f72ec8-e47d-49da-a0d3-1b3ec58ef26c None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:15.848300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f72ec8-e47d-49da-a0d3-1b3ec58ef26c None admin] RBAC: Adding query filter params (type=compute) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:15.848631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f72ec8-e47d-49da-a0d3-1b3ec58ef26c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:15.849998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f72ec8-e47d-49da-a0d3-1b3ec58ef26c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:15.860237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 513/1019] 10.222.0.25 () {56 vars in 1112 bytes} [Mon Sep 26 01:24:15 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:15.861614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcec706-2b16-4426-8052-fee7a6e505b1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:15.916672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:15.965775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcec706-2b16-4426-8052-fee7a6e505b1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:16.001310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba73f39-df74-4c12-923a-e7c5550bd46e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:16.001916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba73f39-df74-4c12-923a-e7c5550bd46e None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:16.002191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba73f39-df74-4c12-923a-e7c5550bd46e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:16.002462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba73f39-df74-4c12-923a-e7c5550bd46e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:16.003121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba73f39-df74-4c12-923a-e7c5550bd46e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:16.004498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba73f39-df74-4c12-923a-e7c5550bd46e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:16.028421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 507/1020] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:24:15 2022] POST /identity/v3/endpoints => generated 334 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:24:16.636000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c33405-105d-4ab5-9847-a8e517252266 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:16.636555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c33405-105d-4ab5-9847-a8e517252266 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:16.636637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c33405-105d-4ab5-9847-a8e517252266 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:16.638047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 514/1021] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:16.645162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7670bf-17fa-4303-90cf-28f97e26cd66 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:16.645440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7670bf-17fa-4303-90cf-28f97e26cd66 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:16.645715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7670bf-17fa-4303-90cf-28f97e26cd66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:16.665744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ef7670bf-17fa-4303-90cf-28f97e26cd66 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:16.666933 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ef7670bf-17fa-4303-90cf-28f97e26cd66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:16.744388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 508/1022] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:16 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:16.983049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f80032-3dfe-4a4f-ad72-dc785c38cd6d None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:16.983455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f80032-3dfe-4a4f-ad72-dc785c38cd6d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:16.983637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f80032-3dfe-4a4f-ad72-dc785c38cd6d None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:16.985010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 515/1023] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:16.990404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f761e07c-e6a4-4f01-92ea-cafe6b78e923 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:16.990695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f761e07c-e6a4-4f01-92ea-cafe6b78e923 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:16.990971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f761e07c-e6a4-4f01-92ea-cafe6b78e923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:17.011860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f761e07c-e6a4-4f01-92ea-cafe6b78e923 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:17.013096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f761e07c-e6a4-4f01-92ea-cafe6b78e923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:17.028468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f80032-3dfe-4a4f-ad72-dc785c38cd6d None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:17.050021 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:17.091407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 509/1024] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:16 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:17.139739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f80032-3dfe-4a4f-ad72-dc785c38cd6d None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:17.151155 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:17.174864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e82e4ce-6647-4318-bbc0-d6dbbb1ebf30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:17.175485 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e82e4ce-6647-4318-bbc0-d6dbbb1ebf30 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:17.175767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e82e4ce-6647-4318-bbc0-d6dbbb1ebf30 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:17.176041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e82e4ce-6647-4318-bbc0-d6dbbb1ebf30 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:17.177186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e82e4ce-6647-4318-bbc0-d6dbbb1ebf30 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:17.178639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e82e4ce-6647-4318-bbc0-d6dbbb1ebf30 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:17.180972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 516/1025] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:17.186464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f761e07c-e6a4-4f01-92ea-cafe6b78e923 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:17.290961 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f761e07c-e6a4-4f01-92ea-cafe6b78e923 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:17.326972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449c4e91-7fdc-41bb-bc3c-8d3b870af254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:17.327629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449c4e91-7fdc-41bb-bc3c-8d3b870af254 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:17.328000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449c4e91-7fdc-41bb-bc3c-8d3b870af254 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:17.328411 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449c4e91-7fdc-41bb-bc3c-8d3b870af254 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:17.329352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449c4e91-7fdc-41bb-bc3c-8d3b870af254 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:17.331526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449c4e91-7fdc-41bb-bc3c-8d3b870af254 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:17.361068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 510/1026] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:24:17 2022] POST /identity/v3/users => generated 285 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:24:17.365445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e82e4ce-6647-4318-bbc0-d6dbbb1ebf30 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:17.477009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e82e4ce-6647-4318-bbc0-d6dbbb1ebf30 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:17.515397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06ec744-e72f-4cd9-9239-5264086d5f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:17.516045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ec744-e72f-4cd9-9239-5264086d5f00 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:17.516327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ec744-e72f-4cd9-9239-5264086d5f00 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:17.516628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ec744-e72f-4cd9-9239-5264086d5f00 None admin] PATH_INFO: `/v3/services/image` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:17.517226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06ec744-e72f-4cd9-9239-5264086d5f00 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:17.518617 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06ec744-e72f-4cd9-9239-5264086d5f00 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-d06ec744-e72f-4cd9-9239-5264086d5f00 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:17.528887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:17.529912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:17.530803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 517/1027] 10.222.0.25 () {56 vars in 1105 bytes} [Mon Sep 26 01:24:17 2022] GET /identity/v3/services/image => generated 86 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:17.535737 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449c4e91-7fdc-41bb-bc3c-8d3b870af254 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:17.561033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:17.647413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449c4e91-7fdc-41bb-bc3c-8d3b870af254 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:17.683053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5883d6-2a0a-4373-9ac4-0e2cbdea4f47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:17.683681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5883d6-2a0a-4373-9ac4-0e2cbdea4f47 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:17.683932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5883d6-2a0a-4373-9ac4-0e2cbdea4f47 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:17.684206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5883d6-2a0a-4373-9ac4-0e2cbdea4f47 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:17.684861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5883d6-2a0a-4373-9ac4-0e2cbdea4f47 None admin] RBAC: Adding query filter params (name=image) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:17.685167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5883d6-2a0a-4373-9ac4-0e2cbdea4f47 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:17.686537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5883d6-2a0a-4373-9ac4-0e2cbdea4f47 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:17.696404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 511/1028] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:17 2022] GET /identity/v3/services?name=image => generated 122 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:17.698769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06ec744-e72f-4cd9-9239-5264086d5f00 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:17.744634 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:17.804699 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06ec744-e72f-4cd9-9239-5264086d5f00 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:17.839343 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ca0560-11c8-459a-b87d-266f3ce5c5c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:17.839873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ca0560-11c8-459a-b87d-266f3ce5c5c8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:17.840154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ca0560-11c8-459a-b87d-266f3ce5c5c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:17.840452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ca0560-11c8-459a-b87d-266f3ce5c5c8 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:17.841077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ca0560-11c8-459a-b87d-266f3ce5c5c8 None admin] RBAC: Adding query filter params (type=image) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:17.841379 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ca0560-11c8-459a-b87d-266f3ce5c5c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:17.842739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ca0560-11c8-459a-b87d-266f3ce5c5c8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:17.852398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 518/1029] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:17 2022] GET /identity/v3/services?type=image => generated 350 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:17.853698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5883d6-2a0a-4373-9ac4-0e2cbdea4f47 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:17.946579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:17.965052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5883d6-2a0a-4373-9ac4-0e2cbdea4f47 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:17.999581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf241d2-cdce-4b9b-9318-b121e37bdb40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:18.000188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf241d2-cdce-4b9b-9318-b121e37bdb40 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:18.000504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf241d2-cdce-4b9b-9318-b121e37bdb40 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:18.000783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf241d2-cdce-4b9b-9318-b121e37bdb40 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:18.001485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf241d2-cdce-4b9b-9318-b121e37bdb40 None admin] RBAC: Adding query filter params (interface=public, service_id=85dba19901fc41f38bf41e9bfe93c5be, region_id=RegionOne) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:18.001796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf241d2-cdce-4b9b-9318-b121e37bdb40 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:18.003167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf241d2-cdce-4b9b-9318-b121e37bdb40 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:18.013480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 512/1030] 10.222.0.25 () {56 vars in 1251 bytes} [Mon Sep 26 01:24:17 2022] GET /identity/v3/endpoints?service_id=85dba19901fc41f38bf41e9bfe93c5be&interface=public®ion_id=RegionOne => generated 194 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:18.015052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ca0560-11c8-459a-b87d-266f3ce5c5c8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:18.123885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ca0560-11c8-459a-b87d-266f3ce5c5c8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:18.131058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:18.159587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed017bb1-f25c-4348-86fc-d2456e56f3d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:18.160194 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed017bb1-f25c-4348-86fc-d2456e56f3d8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:18.160501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed017bb1-f25c-4348-86fc-d2456e56f3d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:18.160771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed017bb1-f25c-4348-86fc-d2456e56f3d8 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:18.161372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed017bb1-f25c-4348-86fc-d2456e56f3d8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:18.162718 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed017bb1-f25c-4348-86fc-d2456e56f3d8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:18.171166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 519/1031] 10.222.0.25 () {56 vars in 1087 bytes} [Mon Sep 26 01:24:18 2022] GET /identity/v3/services => generated 1957 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:24:18.308866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4350ad-d97f-478f-834b-c93683688840 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:18.309265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4350ad-d97f-478f-834b-c93683688840 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:18.309446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4350ad-d97f-478f-834b-c93683688840 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:18.310977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 513/1032] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:18.316638 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb30917-265c-4a39-bbb6-374a03f9fc46 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:18.316918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb30917-265c-4a39-bbb6-374a03f9fc46 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:18.317196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb30917-265c-4a39-bbb6-374a03f9fc46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:18.338461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-2cb30917-265c-4a39-bbb6-374a03f9fc46 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:18.339696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2cb30917-265c-4a39-bbb6-374a03f9fc46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:18.411790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 520/1033] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:18 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:18.680314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4350ad-d97f-478f-834b-c93683688840 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:18.785565 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4350ad-d97f-478f-834b-c93683688840 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:18.817832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb15652-6c79-4a8e-a9af-b6fc774f6440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:18.818459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb15652-6c79-4a8e-a9af-b6fc774f6440 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:18.818728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb15652-6c79-4a8e-a9af-b6fc774f6440 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:18.818995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb15652-6c79-4a8e-a9af-b6fc774f6440 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:18.830897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb15652-6c79-4a8e-a9af-b6fc774f6440 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ebb15652-6c79-4a8e-a9af-b6fc774f6440 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:18.836740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:18.837725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:18.841403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 514/1034] 10.222.0.25 () {56 vars in 1120 bytes} [Mon Sep 26 01:24:18 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:18.843967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb30917-265c-4a39-bbb6-374a03f9fc46 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:18.942979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:18.947116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb30917-265c-4a39-bbb6-374a03f9fc46 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:18.984451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddc2969-07af-45c4-92e4-a284c292a5b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:18.985058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddc2969-07af-45c4-92e4-a284c292a5b8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:18.985337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddc2969-07af-45c4-92e4-a284c292a5b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:18.985610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddc2969-07af-45c4-92e4-a284c292a5b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:18.986269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddc2969-07af-45c4-92e4-a284c292a5b8 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:18.986571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddc2969-07af-45c4-92e4-a284c292a5b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:18.987927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddc2969-07af-45c4-92e4-a284c292a5b8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:19.001122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 521/1035] 10.222.0.25 () {56 vars in 1115 bytes} [Mon Sep 26 01:24:18 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:19.002719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb15652-6c79-4a8e-a9af-b6fc774f6440 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:19.014858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:19.116893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb15652-6c79-4a8e-a9af-b6fc774f6440 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:19.150320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ab7c0-76be-4202-b39f-61e8ebf94364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:19.150940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ab7c0-76be-4202-b39f-61e8ebf94364 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:19.151220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ab7c0-76be-4202-b39f-61e8ebf94364 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:19.151491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ab7c0-76be-4202-b39f-61e8ebf94364 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:19.162219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1ab7c0-76be-4202-b39f-61e8ebf94364 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:19.163965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1ab7c0-76be-4202-b39f-61e8ebf94364 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0c1ab7c0-76be-4202-b39f-61e8ebf94364 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:19.178084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:19.179562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:19.180509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:19.180509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:19.180509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:19.184702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 515/1036] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:19.185956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddc2969-07af-45c4-92e4-a284c292a5b8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:19.289068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddc2969-07af-45c4-92e4-a284c292a5b8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:19.322841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622d9e27-6bca-499a-a39a-9e6f40171325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:19.323447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622d9e27-6bca-499a-a39a-9e6f40171325 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:19.323728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622d9e27-6bca-499a-a39a-9e6f40171325 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:19.324001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622d9e27-6bca-499a-a39a-9e6f40171325 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:19.324661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622d9e27-6bca-499a-a39a-9e6f40171325 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:19.324966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622d9e27-6bca-499a-a39a-9e6f40171325 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:19.326328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622d9e27-6bca-499a-a39a-9e6f40171325 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:19.338890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 522/1037] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:19.344288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ab7c0-76be-4202-b39f-61e8ebf94364 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:19.408127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:19.451910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ab7c0-76be-4202-b39f-61e8ebf94364 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:19.490257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29670f44-605c-473a-b47e-97da74241fb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:19.490877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29670f44-605c-473a-b47e-97da74241fb8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:19.491223 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29670f44-605c-473a-b47e-97da74241fb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:19.491584 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29670f44-605c-473a-b47e-97da74241fb8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:19.499210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29670f44-605c-473a-b47e-97da74241fb8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:19.500688 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29670f44-605c-473a-b47e-97da74241fb8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-29670f44-605c-473a-b47e-97da74241fb8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:24:19.512824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:19.513879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:19.520237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 516/1038] 10.222.0.25 () {56 vars in 1093 bytes} [Mon Sep 26 01:24:19 2022] GET /identity/v3/users/nova => generated 82 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:19.520901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622d9e27-6bca-499a-a39a-9e6f40171325 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:19.549972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:19.626654 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622d9e27-6bca-499a-a39a-9e6f40171325 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:19.634619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c3976f-afc9-4d49-91b4-5235fe1159a4 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:19.634908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c3976f-afc9-4d49-91b4-5235fe1159a4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:19.635183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c3976f-afc9-4d49-91b4-5235fe1159a4 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:19.636666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 517/1039] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:24:19.641588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9ac68c-5026-4247-a853-221ecd41b3d2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:19.641873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9ac68c-5026-4247-a853-221ecd41b3d2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:19.642150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9ac68c-5026-4247-a853-221ecd41b3d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:19.660482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fa9ac68c-5026-4247-a853-221ecd41b3d2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:19.661670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fa9ac68c-5026-4247-a853-221ecd41b3d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:19.665560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c2488b-7c9d-451d-ae7b-4fe75bc1e23a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:19.666366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c2488b-7c9d-451d-ae7b-4fe75bc1e23a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:19.666844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c2488b-7c9d-451d-ae7b-4fe75bc1e23a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:19.667285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c2488b-7c9d-451d-ae7b-4fe75bc1e23a None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:19.668210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c2488b-7c9d-451d-ae7b-4fe75bc1e23a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:19.668709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c2488b-7c9d-451d-ae7b-4fe75bc1e23a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:19.670324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c2488b-7c9d-451d-ae7b-4fe75bc1e23a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:19.691505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 523/1040] 10.222.0.25 () {56 vars in 1133 bytes} [Mon Sep 26 01:24:19 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:19.698098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c2488b-7c9d-451d-ae7b-4fe75bc1e23a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:19.739808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 518/1041] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:19 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:24:19.783231 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:19.803534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c2488b-7c9d-451d-ae7b-4fe75bc1e23a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:19.838476 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0dd0d7-360d-4300-834c-024d9dcf0291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:19.839075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0dd0d7-360d-4300-834c-024d9dcf0291 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:19.839351 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0dd0d7-360d-4300-834c-024d9dcf0291 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:19.839622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0dd0d7-360d-4300-834c-024d9dcf0291 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:19.907100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0dd0d7-360d-4300-834c-024d9dcf0291 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:19.908567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0dd0d7-360d-4300-834c-024d9dcf0291 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:19.987409 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:20.014660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 524/1042] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:19 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:20.015830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9ac68c-5026-4247-a853-221ecd41b3d2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.018442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0dd0d7-360d-4300-834c-024d9dcf0291 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.069935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:20.122905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9ac68c-5026-4247-a853-221ecd41b3d2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:20.133007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0dd0d7-360d-4300-834c-024d9dcf0291 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:20.159772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a855ee-c4ee-4e4c-8c51-76022977666a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:20.160426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a855ee-c4ee-4e4c-8c51-76022977666a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.160721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a855ee-c4ee-4e4c-8c51-76022977666a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.160985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a855ee-c4ee-4e4c-8c51-76022977666a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.167205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d434f-41e9-4ab0-b7b6-595c77f0a078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:20.167828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d434f-41e9-4ab0-b7b6-595c77f0a078 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.168108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d434f-41e9-4ab0-b7b6-595c77f0a078 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.168399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d434f-41e9-4ab0-b7b6-595c77f0a078 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.172790 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a855ee-c4ee-4e4c-8c51-76022977666a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:20.174422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a855ee-c4ee-4e4c-8c51-76022977666a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:20.179527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9d434f-41e9-4ab0-b7b6-595c77f0a078 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-9a9d434f-41e9-4ab0-b7b6-595c77f0a078 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:20.184293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:20.185361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:20.186217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 525/1043] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:20.191353 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d434f-41e9-4ab0-b7b6-595c77f0a078 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a8a855ee-c4ee-4e4c-8c51-76022977666a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:20.192821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:20.193734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:20.199740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:20.199740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:20.199740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:20.199740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 519/1044] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:20.199740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a855ee-c4ee-4e4c-8c51-76022977666a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.243182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:20.306984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a855ee-c4ee-4e4c-8c51-76022977666a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:20.308674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:20.310056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d434f-41e9-4ab0-b7b6-595c77f0a078 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:20.344552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1b0cc4-66f2-4df6-95e3-7d1009fd09b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:20.345073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1b0cc4-66f2-4df6-95e3-7d1009fd09b7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.345352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1b0cc4-66f2-4df6-95e3-7d1009fd09b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.345626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1b0cc4-66f2-4df6-95e3-7d1009fd09b7 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.346263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1b0cc4-66f2-4df6-95e3-7d1009fd09b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:20.346550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7404faff-35b9-4cfe-bcba-371805898a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:20.346679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1b0cc4-66f2-4df6-95e3-7d1009fd09b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:20.347152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404faff-35b9-4cfe-bcba-371805898a3c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.347429 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404faff-35b9-4cfe-bcba-371805898a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.347702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404faff-35b9-4cfe-bcba-371805898a3c None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.347922 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1b0cc4-66f2-4df6-95e3-7d1009fd09b7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:20.348449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7404faff-35b9-4cfe-bcba-371805898a3c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:20.348757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7404faff-35b9-4cfe-bcba-371805898a3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:20.350152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7404faff-35b9-4cfe-bcba-371805898a3c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:20.359481 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 526/1045] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:20.360746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db3310c-9a57-479a-ad20-40e40e3bab29 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.361029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db3310c-9a57-479a-ad20-40e40e3bab29 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.361299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db3310c-9a57-479a-ad20-40e40e3bab29 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.366687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 520/1046] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:20.366852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 527/1047] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:20.366958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1b0cc4-66f2-4df6-95e3-7d1009fd09b7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.367346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db3310c-9a57-479a-ad20-40e40e3bab29 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.463227 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1b0cc4-66f2-4df6-95e3-7d1009fd09b7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:20.469047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db3310c-9a57-479a-ad20-40e40e3bab29 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:20.471322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:20.497785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cac472a-0137-4a41-9773-97685708742c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:20.498403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cac472a-0137-4a41-9773-97685708742c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.498683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cac472a-0137-4a41-9773-97685708742c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.498957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cac472a-0137-4a41-9773-97685708742c None admin] PATH_INFO: `/v3/users/7e596d49f932406dbb57d31e26e44c98` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.502718 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04eaa03-ddb4-444f-bbc2-ab5832083002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:20.503333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04eaa03-ddb4-444f-bbc2-ab5832083002 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.503616 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04eaa03-ddb4-444f-bbc2-ab5832083002 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.503890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04eaa03-ddb4-444f-bbc2-ab5832083002 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.510990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04eaa03-ddb4-444f-bbc2-ab5832083002 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:20.512462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04eaa03-ddb4-444f-bbc2-ab5832083002 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:20.512794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cac472a-0137-4a41-9773-97685708742c None admin] RBAC: Authorizing `identity:get_user(user_id=7e596d49f932406dbb57d31e26e44c98)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:20.514213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cac472a-0137-4a41-9773-97685708742c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:20.515821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 521/1048] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/users/7e596d49f932406dbb57d31e26e44c98 => generated 285 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:20.516977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561861e-7a0a-463b-84d2-92e1094418d4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.517256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561861e-7a0a-463b-84d2-92e1094418d4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.517522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561861e-7a0a-463b-84d2-92e1094418d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-d04eaa03-ddb4-444f-bbc2-ab5832083002 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:24:20.524174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:20.525216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:20.526167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:24:20.526167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:24:20.526167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:24:20.526167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:24:20.526167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:24:20.526167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:24:20.526167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:20.526167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:20.526167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 528/1049] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:20.526167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04eaa03-ddb4-444f-bbc2-ab5832083002 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.540051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b561861e-7a0a-463b-84d2-92e1094418d4 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:20.541266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b561861e-7a0a-463b-84d2-92e1094418d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:20.555175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:20.608884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 522/1050] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:20 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:20.610086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b561861e-7a0a-463b-84d2-92e1094418d4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.629502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04eaa03-ddb4-444f-bbc2-ab5832083002 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:20.670730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656c5603-af41-4a98-b47d-b7a26d93229f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:20.671360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656c5603-af41-4a98-b47d-b7a26d93229f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.671621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656c5603-af41-4a98-b47d-b7a26d93229f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.671892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656c5603-af41-4a98-b47d-b7a26d93229f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.709426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:20.719957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b561861e-7a0a-463b-84d2-92e1094418d4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:20.751139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656c5603-af41-4a98-b47d-b7a26d93229f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:20.752609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656c5603-af41-4a98-b47d-b7a26d93229f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:20.756387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fee334-db91-4067-9657-bf211b266aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:20.757585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fee334-db91-4067-9657-bf211b266aa7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.757861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fee334-db91-4067-9657-bf211b266aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.758132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fee334-db91-4067-9657-bf211b266aa7 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.758817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fee334-db91-4067-9657-bf211b266aa7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:20.759120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fee334-db91-4067-9657-bf211b266aa7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:20.760561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fee334-db91-4067-9657-bf211b266aa7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:20.772215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 523/1051] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:20.778256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fee334-db91-4067-9657-bf211b266aa7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.791109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:20.864843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 529/1052] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:20.870947 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656c5603-af41-4a98-b47d-b7a26d93229f None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.887781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fee334-db91-4067-9657-bf211b266aa7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:20.922104 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2b2fe2-bc7c-4804-b5c8-da3a3426c809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:20.922702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2b2fe2-bc7c-4804-b5c8-da3a3426c809 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:20.922981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2b2fe2-bc7c-4804-b5c8-da3a3426c809 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:20.923262 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2b2fe2-bc7c-4804-b5c8-da3a3426c809 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:20.923973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2b2fe2-bc7c-4804-b5c8-da3a3426c809 None admin] RBAC: Adding query filter params (role.id=fe72ae0ebcab44b69a769caf591d6eda, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=381ad7c239f444da90369dc58779b86c) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:20.924285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2b2fe2-bc7c-4804-b5c8-da3a3426c809 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:20.925744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2b2fe2-bc7c-4804-b5c8-da3a3426c809 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:20.931019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 524/1053] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/role_assignments?user.id=381ad7c239f444da90369dc58779b86c&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=fe72ae0ebcab44b69a769caf591d6eda => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:20.931500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:20.931962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2b2fe2-bc7c-4804-b5c8-da3a3426c809 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:20.992225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656c5603-af41-4a98-b47d-b7a26d93229f None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:21.021086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:21.037312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f5648-2fe5-4bbc-9dd4-86b43032ad02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:21.037966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f5648-2fe5-4bbc-9dd4-86b43032ad02 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:21.038251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f5648-2fe5-4bbc-9dd4-86b43032ad02 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:21.038526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f5648-2fe5-4bbc-9dd4-86b43032ad02 None admin] PATH_INFO: `/v3/projects/6f6e7af0f4294958b037d48b7b06f959` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:21.039729 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563f5648-2fe5-4bbc-9dd4-86b43032ad02 None admin] RBAC: Authorizing `identity:get_project(project_id=6f6e7af0f4294958b037d48b7b06f959)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:21.041226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563f5648-2fe5-4bbc-9dd4-86b43032ad02 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:21.050831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 530/1054] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/projects/6f6e7af0f4294958b037d48b7b06f959 => generated 311 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:21.053209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f5648-2fe5-4bbc-9dd4-86b43032ad02 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:21.059444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2b2fe2-bc7c-4804-b5c8-da3a3426c809 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:21.099022 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5964cc-e5bd-4f6d-9994-53dba146fa2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:21.099655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5964cc-e5bd-4f6d-9994-53dba146fa2f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:21.099939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5964cc-e5bd-4f6d-9994-53dba146fa2f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:21.100205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5964cc-e5bd-4f6d-9994-53dba146fa2f None admin] PATH_INFO: `/v3/services/image` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:21.101215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5964cc-e5bd-4f6d-9994-53dba146fa2f None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:21.102668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5964cc-e5bd-4f6d-9994-53dba146fa2f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-8e5964cc-e5bd-4f6d-9994-53dba146fa2f None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:21.113107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:21.114143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:21.115047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 525/1055] 10.222.0.25 () {56 vars in 1105 bytes} [Mon Sep 26 01:24:20 2022] GET /identity/v3/services/image => generated 86 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:21.120628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5964cc-e5bd-4f6d-9994-53dba146fa2f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:21.168369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f5648-2fe5-4bbc-9dd4-86b43032ad02 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:21.198524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:21.206009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40a4e35-082b-40e0-b440-6305a40465fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:21.206616 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40a4e35-082b-40e0-b440-6305a40465fb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:21.207161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40a4e35-082b-40e0-b440-6305a40465fb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:21.207434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40a4e35-082b-40e0-b440-6305a40465fb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:21.208193 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40a4e35-082b-40e0-b440-6305a40465fb None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=6f6e7af0f4294958b037d48b7b06f959, user.id=7e596d49f932406dbb57d31e26e44c98) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:21.208538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40a4e35-082b-40e0-b440-6305a40465fb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:21.209974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40a4e35-082b-40e0-b440-6305a40465fb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:21.216053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 531/1056] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:24:21 2022] GET /identity/v3/role_assignments?user.id=7e596d49f932406dbb57d31e26e44c98&scope.project.id=6f6e7af0f4294958b037d48b7b06f959&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:21.240860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5964cc-e5bd-4f6d-9994-53dba146fa2f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:21.280269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad097bba-7372-40e1-add0-8a1735a1da1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:21.281219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad097bba-7372-40e1-add0-8a1735a1da1d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:21.281672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad097bba-7372-40e1-add0-8a1735a1da1d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:21.282088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad097bba-7372-40e1-add0-8a1735a1da1d None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:21.282872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad097bba-7372-40e1-add0-8a1735a1da1d None admin] RBAC: Adding query filter params (name=image) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:21.283322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad097bba-7372-40e1-add0-8a1735a1da1d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:21.284889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad097bba-7372-40e1-add0-8a1735a1da1d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:21.291920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 526/1057] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:21 2022] GET /identity/v3/services?name=image => generated 122 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:21.300389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40a4e35-082b-40e0-b440-6305a40465fb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:21.417869 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40a4e35-082b-40e0-b440-6305a40465fb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:21.452619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4b0b42-caad-4073-b202-95f49a9d3cd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:21.453220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b0b42-caad-4073-b202-95f49a9d3cd2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:21.453495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b0b42-caad-4073-b202-95f49a9d3cd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:21.453763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b0b42-caad-4073-b202-95f49a9d3cd2 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:21.454381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4b0b42-caad-4073-b202-95f49a9d3cd2 None admin] RBAC: Adding query filter params (type=image) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:21.454683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4b0b42-caad-4073-b202-95f49a9d3cd2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:21.456034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4b0b42-caad-4073-b202-95f49a9d3cd2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:21.466284 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 532/1058] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:21 2022] GET /identity/v3/services?type=image => generated 350 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:21.467550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad097bba-7372-40e1-add0-8a1735a1da1d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:21.470470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:21.572122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad097bba-7372-40e1-add0-8a1735a1da1d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:21.607620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede0b92a-dc7a-4624-b364-da625cae198d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:21.608397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede0b92a-dc7a-4624-b364-da625cae198d None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:21.608779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede0b92a-dc7a-4624-b364-da625cae198d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:21.609152 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede0b92a-dc7a-4624-b364-da625cae198d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:21.609934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede0b92a-dc7a-4624-b364-da625cae198d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:21.611397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede0b92a-dc7a-4624-b364-da625cae198d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:21.640113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 527/1059] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:24:21 2022] POST /identity/v3/endpoints => generated 327 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:24:23.114871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ebdd83-a667-4170-96a3-2ed22b7ed945 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:23.115435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ebdd83-a667-4170-96a3-2ed22b7ed945 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:23.115435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ebdd83-a667-4170-96a3-2ed22b7ed945 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:23.117622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 533/1060] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:23.122742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539b5ac-06da-4e13-9367-5540bef71353 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:23.123159 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539b5ac-06da-4e13-9367-5540bef71353 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:23.123345 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539b5ac-06da-4e13-9367-5540bef71353 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:23.142149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8539b5ac-06da-4e13-9367-5540bef71353 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:23.143341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8539b5ac-06da-4e13-9367-5540bef71353 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:23.198470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:23.220993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 528/1061] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:23.403458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ac68e5-e4fd-46a3-8bdd-2695619a9822 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:23.403644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ac68e5-e4fd-46a3-8bdd-2695619a9822 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:23.403912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ac68e5-e4fd-46a3-8bdd-2695619a9822 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:23.405388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 534/1062] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:23.410365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c2196-9245-41d1-bd5c-1b77c77fa71a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:23.410653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c2196-9245-41d1-bd5c-1b77c77fa71a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:23.410930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c2196-9245-41d1-bd5c-1b77c77fa71a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:23.429775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-314c2196-9245-41d1-bd5c-1b77c77fa71a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:23.430944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-314c2196-9245-41d1-bd5c-1b77c77fa71a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:23.492214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ac68e5-e4fd-46a3-8bdd-2695619a9822 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:23.504383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 529/1063] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:23.552857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:23.598418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ac68e5-e4fd-46a3-8bdd-2695619a9822 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:23.633254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f26357a-42e3-4db9-a141-67207f136992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:23.633880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f26357a-42e3-4db9-a141-67207f136992 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:23.634161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f26357a-42e3-4db9-a141-67207f136992 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:23.634437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f26357a-42e3-4db9-a141-67207f136992 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:23.644454 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f26357a-42e3-4db9-a141-67207f136992 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-5f26357a-42e3-4db9-a141-67207f136992 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:23.649911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:23.650931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:24:23.651862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:23.656019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 535/1064] 10.222.0.25 () {56 vars in 1120 bytes} [Mon Sep 26 01:24:23 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:23.657249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314c2196-9245-41d1-bd5c-1b77c77fa71a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:23.774309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314c2196-9245-41d1-bd5c-1b77c77fa71a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:23.781482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f26357a-42e3-4db9-a141-67207f136992 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:23.790591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:23.814488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53135e1-d624-470d-a124-ab615a85ceaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:23.815082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53135e1-d624-470d-a124-ab615a85ceaf None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:23.815363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53135e1-d624-470d-a124-ab615a85ceaf None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:23.815630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53135e1-d624-470d-a124-ab615a85ceaf None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:23.816296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53135e1-d624-470d-a124-ab615a85ceaf None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:23.816625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53135e1-d624-470d-a124-ab615a85ceaf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:23.817984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53135e1-d624-470d-a124-ab615a85ceaf None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:23.827514 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 530/1065] 10.222.0.25 () {56 vars in 1115 bytes} [Mon Sep 26 01:24:23 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:23.833344 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53135e1-d624-470d-a124-ab615a85ceaf None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:23.885828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f26357a-42e3-4db9-a141-67207f136992 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:23.904819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:23.922792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17de712d-5705-4642-b9e9-1737b410c267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:23.923345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17de712d-5705-4642-b9e9-1737b410c267 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:23.923617 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17de712d-5705-4642-b9e9-1737b410c267 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:23.923886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17de712d-5705-4642-b9e9-1737b410c267 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:23.933469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53135e1-d624-470d-a124-ab615a85ceaf None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:23.934130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17de712d-5705-4642-b9e9-1737b410c267 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-17de712d-5705-4642-b9e9-1737b410c267 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:23.938835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:23.939786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:23.940685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 536/1066] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:24:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:23.941865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2b8db-2572-482b-9909-407d14dfada7 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:23.942512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2b8db-2572-482b-9909-407d14dfada7 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:23.942845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2b8db-2572-482b-9909-407d14dfada7 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:23.945341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 537/1067] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:23.946205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b2b8db-2572-482b-9909-407d14dfada7 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:23.970876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8db75ce-b2f0-499f-bed4-b08a6003f005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:23.971494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db75ce-b2f0-499f-bed4-b08a6003f005 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:23.971767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db75ce-b2f0-499f-bed4-b08a6003f005 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:23.972041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db75ce-b2f0-499f-bed4-b08a6003f005 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:23.982638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8db75ce-b2f0-499f-bed4-b08a6003f005 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:23.984069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8db75ce-b2f0-499f-bed4-b08a6003f005 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-c8db75ce-b2f0-499f-bed4-b08a6003f005 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:23.998703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:23.999879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:24.001028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:24.001028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:24.001028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:24.004783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 531/1068] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:24.006164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09203139-9b67-4355-b4b6-c5cd8a292b69 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.006468 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09203139-9b67-4355-b4b6-c5cd8a292b69 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.006839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09203139-9b67-4355-b4b6-c5cd8a292b69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.024417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-09203139-9b67-4355-b4b6-c5cd8a292b69 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:24.025616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-09203139-9b67-4355-b4b6-c5cd8a292b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:24.049801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b2b8db-2572-482b-9909-407d14dfada7 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:24.049960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:24.084280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69da3a98-6d64-4083-93e7-d938452a2e9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:24.085641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69da3a98-6d64-4083-93e7-d938452a2e9c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.086064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69da3a98-6d64-4083-93e7-d938452a2e9c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.086342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69da3a98-6d64-4083-93e7-d938452a2e9c None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.087032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69da3a98-6d64-4083-93e7-d938452a2e9c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:24.087339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69da3a98-6d64-4083-93e7-d938452a2e9c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:24.088722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69da3a98-6d64-4083-93e7-d938452a2e9c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:24.095127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 532/1069] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:24.095794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09203139-9b67-4355-b4b6-c5cd8a292b69 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:24.111296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 538/1070] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:24:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:24.113011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69da3a98-6d64-4083-93e7-d938452a2e9c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:24.157650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:24.203498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09203139-9b67-4355-b4b6-c5cd8a292b69 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:24.222538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69da3a98-6d64-4083-93e7-d938452a2e9c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:24.239344 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb5cca1-79bd-422d-86ef-e22372bc7d89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:24.239946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb5cca1-79bd-422d-86ef-e22372bc7d89 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.240226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb5cca1-79bd-422d-86ef-e22372bc7d89 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.240535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb5cca1-79bd-422d-86ef-e22372bc7d89 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.241383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb5cca1-79bd-422d-86ef-e22372bc7d89 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:24.241765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb5cca1-79bd-422d-86ef-e22372bc7d89 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:24.243139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb5cca1-79bd-422d-86ef-e22372bc7d89 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:24.260369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 533/1071] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:24.260542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b5138-b8e8-412e-ad0c-5a0da930acd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:24.261199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b5138-b8e8-412e-ad0c-5a0da930acd8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.261199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b5138-b8e8-412e-ad0c-5a0da930acd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.261461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b5138-b8e8-412e-ad0c-5a0da930acd8 None admin] PATH_INFO: `/v3/users/7e596d49f932406dbb57d31e26e44c98` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.261867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb5cca1-79bd-422d-86ef-e22372bc7d89 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:24.262590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95b5138-b8e8-412e-ad0c-5a0da930acd8 None admin] RBAC: Authorizing `identity:get_user(user_id=7e596d49f932406dbb57d31e26e44c98)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:24.264070 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95b5138-b8e8-412e-ad0c-5a0da930acd8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:24.266189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 539/1072] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:24 2022] GET /identity/v3/users/7e596d49f932406dbb57d31e26e44c98 => generated 285 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:24.271870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b5138-b8e8-412e-ad0c-5a0da930acd8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:24.283972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:24.368611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb5cca1-79bd-422d-86ef-e22372bc7d89 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:24.379526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b5138-b8e8-412e-ad0c-5a0da930acd8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:24.388889 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:24.402846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4bafb6-cb20-48c7-9270-e4a02707c489 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:24.403477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4bafb6-cb20-48c7-9270-e4a02707c489 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.403758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4bafb6-cb20-48c7-9270-e4a02707c489 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.404032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4bafb6-cb20-48c7-9270-e4a02707c489 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.411476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4bafb6-cb20-48c7-9270-e4a02707c489 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:24.413050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4bafb6-cb20-48c7-9270-e4a02707c489 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:24.415446 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0ad100-e5b2-43f6-8901-2e8615cbb390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:24.416060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0ad100-e5b2-43f6-8901-2e8615cbb390 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.416333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0ad100-e5b2-43f6-8901-2e8615cbb390 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.416630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0ad100-e5b2-43f6-8901-2e8615cbb390 None admin] PATH_INFO: `/v3/projects/6f6e7af0f4294958b037d48b7b06f959` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.417732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0ad100-e5b2-43f6-8901-2e8615cbb390 None admin] RBAC: Authorizing `identity:get_project(project_id=6f6e7af0f4294958b037d48b7b06f959)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:24.419127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0ad100-e5b2-43f6-8901-2e8615cbb390 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:24.420751 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 540/1073] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:24:24 2022] GET /identity/v3/projects/6f6e7af0f4294958b037d48b7b06f959 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:24.421504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0ad100-e5b2-43f6-8901-2e8615cbb390 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0c4bafb6-cb20-48c7-9270-e4a02707c489 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:24:24.424069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:24.425060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:24.426060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 534/1074] 10.222.0.25 () {56 vars in 1093 bytes} [Mon Sep 26 01:24:24 2022] GET /identity/v3/users/nova => generated 82 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:24.427314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4bafb6-cb20-48c7-9270-e4a02707c489 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:24.504835 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:24.523943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0ad100-e5b2-43f6-8901-2e8615cbb390 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:24.528146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4bafb6-cb20-48c7-9270-e4a02707c489 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:24.557386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc23e3-86a0-4ae1-bc4e-ebdcb33ddcd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:24.558020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc23e3-86a0-4ae1-bc4e-ebdcb33ddcd9 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.558306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc23e3-86a0-4ae1-bc4e-ebdcb33ddcd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.558584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc23e3-86a0-4ae1-bc4e-ebdcb33ddcd9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.561187 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea91035f-b382-44a1-9088-966a1027a70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:24.561835 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea91035f-b382-44a1-9088-966a1027a70c None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.562107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea91035f-b382-44a1-9088-966a1027a70c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.562377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea91035f-b382-44a1-9088-966a1027a70c None admin] PATH_INFO: `/v3/projects/6f6e7af0f4294958b037d48b7b06f959/users/7e596d49f932406dbb57d31e26e44c98/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.564249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea91035f-b382-44a1-9088-966a1027a70c None admin] RBAC: Authorizing `identity:create_grant(project_id=6f6e7af0f4294958b037d48b7b06f959, user_id=7e596d49f932406dbb57d31e26e44c98, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:24.565711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea91035f-b382-44a1-9088-966a1027a70c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:24.579476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 535/1075] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:24:24 2022] PUT /identity/v3/projects/6f6e7af0f4294958b037d48b7b06f959/users/7e596d49f932406dbb57d31e26e44c98/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:24:24.580524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea91035f-b382-44a1-9088-966a1027a70c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:24.609803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:24.637777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fc23e3-86a0-4ae1-bc4e-ebdcb33ddcd9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:24.639265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fc23e3-86a0-4ae1-bc4e-ebdcb33ddcd9 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:24.700446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea91035f-b382-44a1-9088-966a1027a70c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:24.738106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ad6012-2893-4308-8e0a-0acb4330f109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:24.738718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ad6012-2893-4308-8e0a-0acb4330f109 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.738995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ad6012-2893-4308-8e0a-0acb4330f109 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.739264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ad6012-2893-4308-8e0a-0acb4330f109 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.740005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ad6012-2893-4308-8e0a-0acb4330f109 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:24.740310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ad6012-2893-4308-8e0a-0acb4330f109 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:24.742475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ad6012-2893-4308-8e0a-0acb4330f109 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:24.751520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:24.765056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 536/1076] 10.222.0.25 () {56 vars in 1133 bytes} [Mon Sep 26 01:24:24 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:24.771652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ad6012-2893-4308-8e0a-0acb4330f109 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:24.773612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 541/1077] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:24.779330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc23e3-86a0-4ae1-bc4e-ebdcb33ddcd9 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:24.867862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:24.892226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ad6012-2893-4308-8e0a-0acb4330f109 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:24.893133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc23e3-86a0-4ae1-bc4e-ebdcb33ddcd9 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:24.929753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e86b7e-a131-4ffc-aa92-8f0967ff685f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:24.930662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e86b7e-a131-4ffc-aa92-8f0967ff685f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.931096 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e86b7e-a131-4ffc-aa92-8f0967ff685f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.931542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e86b7e-a131-4ffc-aa92-8f0967ff685f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.934843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e8b64d-737c-407e-9504-b3c79ce4c86e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:24.935467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e8b64d-737c-407e-9504-b3c79ce4c86e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:24.935749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e8b64d-737c-407e-9504-b3c79ce4c86e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:24.936020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e8b64d-737c-407e-9504-b3c79ce4c86e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:24.943020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e86b7e-a131-4ffc-aa92-8f0967ff685f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:24.945012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e86b7e-a131-4ffc-aa92-8f0967ff685f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:24.946677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e8b64d-737c-407e-9504-b3c79ce4c86e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:24.948117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e8b64d-737c-407e-9504-b3c79ce4c86e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-e8e86b7e-a131-4ffc-aa92-8f0967ff685f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:24.961076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:24.962261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:24.963286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:24.963286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:24.963286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-83e8b64d-737c-407e-9504-b3c79ce4c86e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:24.963698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:24.964840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:24.965779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:24.965779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:24.965779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:24.965779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 537/1078] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:24.967033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 542/1079] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:24.970428 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e8b64d-737c-407e-9504-b3c79ce4c86e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:24.977461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e86b7e-a131-4ffc-aa92-8f0967ff685f None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:25.022755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:25.086150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e8b64d-737c-407e-9504-b3c79ce4c86e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:25.087289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e86b7e-a131-4ffc-aa92-8f0967ff685f None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:25.115127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:25.123177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eb00c9-64d7-42d0-887c-e0e7076f6b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:25.123590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe14c30-c8c2-417b-8bbc-201830450282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:25.123809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eb00c9-64d7-42d0-887c-e0e7076f6b53 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:25.124064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eb00c9-64d7-42d0-887c-e0e7076f6b53 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:25.124235 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe14c30-c8c2-417b-8bbc-201830450282 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:25.124464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eb00c9-64d7-42d0-887c-e0e7076f6b53 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:25.124661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe14c30-c8c2-417b-8bbc-201830450282 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:25.124930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe14c30-c8c2-417b-8bbc-201830450282 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:25.125026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68eb00c9-64d7-42d0-887c-e0e7076f6b53 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:25.125282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68eb00c9-64d7-42d0-887c-e0e7076f6b53 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:25.125571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe14c30-c8c2-417b-8bbc-201830450282 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:25.125874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe14c30-c8c2-417b-8bbc-201830450282 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:25.126654 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68eb00c9-64d7-42d0-887c-e0e7076f6b53 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:25.127216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe14c30-c8c2-417b-8bbc-201830450282 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:25.140466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 538/1080] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:25.140643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 543/1081] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:25.145932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eb00c9-64d7-42d0-887c-e0e7076f6b53 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:25.258516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eb00c9-64d7-42d0-887c-e0e7076f6b53 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:25.269175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:25.299281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e18b80-6eab-4252-b98e-21c463a42a71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:25.299840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e18b80-6eab-4252-b98e-21c463a42a71 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:25.300118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e18b80-6eab-4252-b98e-21c463a42a71 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:25.300419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e18b80-6eab-4252-b98e-21c463a42a71 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:25.308497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e18b80-6eab-4252-b98e-21c463a42a71 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:25.309960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e18b80-6eab-4252-b98e-21c463a42a71 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d7e18b80-6eab-4252-b98e-21c463a42a71 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:24:25.321991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:25.322957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:25.325062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:24:25.325062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:24:25.325062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:24:25.325062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:24:25.325062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:24:25.325062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:24:25.325062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:25.325062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:25.325062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 539/1082] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:24:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:25.329079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe14c30-c8c2-417b-8bbc-201830450282 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:25.437332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe14c30-c8c2-417b-8bbc-201830450282 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:25.474147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ec99ad-6b22-4073-b7cb-7ff5876f1a9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:25.474773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ec99ad-6b22-4073-b7cb-7ff5876f1a9c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:25.475052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ec99ad-6b22-4073-b7cb-7ff5876f1a9c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:25.475956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ec99ad-6b22-4073-b7cb-7ff5876f1a9c None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:25.476899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ec99ad-6b22-4073-b7cb-7ff5876f1a9c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:25.477209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ec99ad-6b22-4073-b7cb-7ff5876f1a9c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:25.478779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ec99ad-6b22-4073-b7cb-7ff5876f1a9c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:25.491486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 544/1083] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:24:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:25.497541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e18b80-6eab-4252-b98e-21c463a42a71 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:25.596711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e18b80-6eab-4252-b98e-21c463a42a71 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:25.629425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91b95f9-514d-4e81-be60-ae566f41ba2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:25.630061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b95f9-514d-4e81-be60-ae566f41ba2f None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:25.630340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b95f9-514d-4e81-be60-ae566f41ba2f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:25.630613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b95f9-514d-4e81-be60-ae566f41ba2f None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/381ad7c239f444da90369dc58779b86c/roles/fe72ae0ebcab44b69a769caf591d6eda` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:25.633362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91b95f9-514d-4e81-be60-ae566f41ba2f None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=381ad7c239f444da90369dc58779b86c, role_id=fe72ae0ebcab44b69a769caf591d6eda)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:25.634794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91b95f9-514d-4e81-be60-ae566f41ba2f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:25.647305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 540/1084] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:24:25 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/381ad7c239f444da90369dc58779b86c/roles/fe72ae0ebcab44b69a769caf591d6eda => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:24:26.751065 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4734a03f-7be3-4fd0-8b0d-18c9e753e408 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:26.751625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4734a03f-7be3-4fd0-8b0d-18c9e753e408 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:26.751625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4734a03f-7be3-4fd0-8b0d-18c9e753e408 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:26.753114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 545/1085] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:26.758746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bf1744-a603-46c1-8687-31423347177d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:26.759170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bf1744-a603-46c1-8687-31423347177d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:26.759290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bf1744-a603-46c1-8687-31423347177d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:26.777150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a7bf1744-a603-46c1-8687-31423347177d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:26.778324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a7bf1744-a603-46c1-8687-31423347177d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:26.800893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:26.846316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 541/1086] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:27.113013 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4734a03f-7be3-4fd0-8b0d-18c9e753e408 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:27.162314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:27.215073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4734a03f-7be3-4fd0-8b0d-18c9e753e408 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:27.248555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a9216f-919e-486c-ab1d-d3997a9fffc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:27.249171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a9216f-919e-486c-ab1d-d3997a9fffc6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:27.249446 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a9216f-919e-486c-ab1d-d3997a9fffc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:27.249713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a9216f-919e-486c-ab1d-d3997a9fffc6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:27.260061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a9216f-919e-486c-ab1d-d3997a9fffc6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-40a9216f-919e-486c-ab1d-d3997a9fffc6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:27.264846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:27.265948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:27.266992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:27.268498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 546/1087] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:24:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:27.271881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bf1744-a603-46c1-8687-31423347177d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:27.368066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bf1744-a603-46c1-8687-31423347177d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:27.402124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc937d1-65bd-4346-bff4-21476790b1dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:27.402726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc937d1-65bd-4346-bff4-21476790b1dd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:27.403008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc937d1-65bd-4346-bff4-21476790b1dd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:27.403280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc937d1-65bd-4346-bff4-21476790b1dd None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:27.403939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc937d1-65bd-4346-bff4-21476790b1dd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:27.404240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc937d1-65bd-4346-bff4-21476790b1dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:27.405637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc937d1-65bd-4346-bff4-21476790b1dd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:27.414107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 542/1088] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:24:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:27.419960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a9216f-919e-486c-ab1d-d3997a9fffc6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:27.518391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a9216f-919e-486c-ab1d-d3997a9fffc6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:27.528214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:27.552658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a969c5-07fd-473f-8c8d-702af7139489 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:27.553283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a969c5-07fd-473f-8c8d-702af7139489 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:27.553594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a969c5-07fd-473f-8c8d-702af7139489 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:27.553867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a969c5-07fd-473f-8c8d-702af7139489 None admin] PATH_INFO: `/v3/users/7e596d49f932406dbb57d31e26e44c98` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:27.554974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a969c5-07fd-473f-8c8d-702af7139489 None admin] RBAC: Authorizing `identity:get_user(user_id=7e596d49f932406dbb57d31e26e44c98)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:27.556422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a969c5-07fd-473f-8c8d-702af7139489 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:27.557975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 547/1089] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:27 2022] GET /identity/v3/users/7e596d49f932406dbb57d31e26e44c98 => generated 285 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:27.563795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc937d1-65bd-4346-bff4-21476790b1dd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:27.586265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:27.665690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc937d1-65bd-4346-bff4-21476790b1dd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:27.703488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faabd9a7-49d2-4e88-8bac-911ddf562557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:27.704086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faabd9a7-49d2-4e88-8bac-911ddf562557 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:27.704385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faabd9a7-49d2-4e88-8bac-911ddf562557 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:27.704664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faabd9a7-49d2-4e88-8bac-911ddf562557 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:27.772013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faabd9a7-49d2-4e88-8bac-911ddf562557 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:27.773791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faabd9a7-49d2-4e88-8bac-911ddf562557 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:27.780321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2d4e01-274a-4a59-bb78-f31d7bde5439 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:27.780790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2d4e01-274a-4a59-bb78-f31d7bde5439 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:27.780883 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2d4e01-274a-4a59-bb78-f31d7bde5439 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:27.782336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 548/1090] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:27.786993 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74dca29-7199-4e4f-92ed-32d6d10bb15e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:27.787314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74dca29-7199-4e4f-92ed-32d6d10bb15e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:27.787553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74dca29-7199-4e4f-92ed-32d6d10bb15e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:27.794964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:27.807417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e74dca29-7199-4e4f-92ed-32d6d10bb15e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:27.808642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e74dca29-7199-4e4f-92ed-32d6d10bb15e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:27.875448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 549/1091] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:24:27.882690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 543/1092] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:27.885352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74dca29-7199-4e4f-92ed-32d6d10bb15e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:27.981086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74dca29-7199-4e4f-92ed-32d6d10bb15e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:27.988163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:28.014796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a29eac-4a1c-4be1-b6b9-7553e24e17d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:28.015419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a29eac-4a1c-4be1-b6b9-7553e24e17d7 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:28.015693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a29eac-4a1c-4be1-b6b9-7553e24e17d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:28.015958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a29eac-4a1c-4be1-b6b9-7553e24e17d7 None admin] PATH_INFO: `/v3/projects/6f6e7af0f4294958b037d48b7b06f959` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:28.017073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a29eac-4a1c-4be1-b6b9-7553e24e17d7 None admin] RBAC: Authorizing `identity:get_project(project_id=6f6e7af0f4294958b037d48b7b06f959)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:28.018482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a29eac-4a1c-4be1-b6b9-7553e24e17d7 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:28.020246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 550/1093] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:24:27 2022] GET /identity/v3/projects/6f6e7af0f4294958b037d48b7b06f959 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:28.026230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faabd9a7-49d2-4e88-8bac-911ddf562557 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:28.128412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faabd9a7-49d2-4e88-8bac-911ddf562557 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:28.142627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a29eac-4a1c-4be1-b6b9-7553e24e17d7 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:28.149537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:28.164412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f8d70-46fb-4eb1-a3c8-1ae4c83bc3ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:28.165008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f8d70-46fb-4eb1-a3c8-1ae4c83bc3ae None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:28.165285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f8d70-46fb-4eb1-a3c8-1ae4c83bc3ae None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:28.165557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249f8d70-46fb-4eb1-a3c8-1ae4c83bc3ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:28.166276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249f8d70-46fb-4eb1-a3c8-1ae4c83bc3ae None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=6f6e7af0f4294958b037d48b7b06f959, user.id=7e596d49f932406dbb57d31e26e44c98) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:28.166578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249f8d70-46fb-4eb1-a3c8-1ae4c83bc3ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:28.168013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249f8d70-46fb-4eb1-a3c8-1ae4c83bc3ae None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:28.173697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 544/1094] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:24:28 2022] GET /identity/v3/role_assignments?user.id=7e596d49f932406dbb57d31e26e44c98&scope.project.id=6f6e7af0f4294958b037d48b7b06f959&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:28.249287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a29eac-4a1c-4be1-b6b9-7553e24e17d7 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:28.284990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926f878d-e7ab-44ca-bfbb-28b587f67e5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:28.285628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f878d-e7ab-44ca-bfbb-28b587f67e5a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:28.285910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f878d-e7ab-44ca-bfbb-28b587f67e5a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:28.286184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f878d-e7ab-44ca-bfbb-28b587f67e5a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:28.298677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926f878d-e7ab-44ca-bfbb-28b587f67e5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-926f878d-e7ab-44ca-bfbb-28b587f67e5a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:28.303667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:28.308762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:28.309943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 551/1095] 10.222.0.25 () {56 vars in 1120 bytes} [Mon Sep 26 01:24:28 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:28.311169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f8d70-46fb-4eb1-a3c8-1ae4c83bc3ae None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:28.418261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249f8d70-46fb-4eb1-a3c8-1ae4c83bc3ae None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:28.451049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d9a9e8-697d-4db5-85d0-7bb71daf91b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:28.451640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d9a9e8-697d-4db5-85d0-7bb71daf91b2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:28.451915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d9a9e8-697d-4db5-85d0-7bb71daf91b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:28.452182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d9a9e8-697d-4db5-85d0-7bb71daf91b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:28.452869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d9a9e8-697d-4db5-85d0-7bb71daf91b2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:28.453169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d9a9e8-697d-4db5-85d0-7bb71daf91b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:28.454518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d9a9e8-697d-4db5-85d0-7bb71daf91b2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:28.463063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 545/1096] 10.222.0.25 () {56 vars in 1115 bytes} [Mon Sep 26 01:24:28 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:28.468838 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926f878d-e7ab-44ca-bfbb-28b587f67e5a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:28.502809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:28.569237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926f878d-e7ab-44ca-bfbb-28b587f67e5a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:28.602756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65ed305-2950-4e66-b9f6-cd3c5e2f897d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:28.603370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65ed305-2950-4e66-b9f6-cd3c5e2f897d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:28.603643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65ed305-2950-4e66-b9f6-cd3c5e2f897d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:28.603908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65ed305-2950-4e66-b9f6-cd3c5e2f897d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:28.614080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65ed305-2950-4e66-b9f6-cd3c5e2f897d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:28.615505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65ed305-2950-4e66-b9f6-cd3c5e2f897d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-d65ed305-2950-4e66-b9f6-cd3c5e2f897d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:28.629963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:28.631094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:28.632229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:28.632229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:28.632229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:28.632229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 552/1097] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:28.638438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d9a9e8-697d-4db5-85d0-7bb71daf91b2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:28.685632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:28.739900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d9a9e8-697d-4db5-85d0-7bb71daf91b2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:28.773353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2daa5a2a-2959-4717-9214-ef02fc4c8b68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:28.773959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daa5a2a-2959-4717-9214-ef02fc4c8b68 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:28.774139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daa5a2a-2959-4717-9214-ef02fc4c8b68 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:28.774413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daa5a2a-2959-4717-9214-ef02fc4c8b68 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:28.775052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2daa5a2a-2959-4717-9214-ef02fc4c8b68 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:28.775359 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2daa5a2a-2959-4717-9214-ef02fc4c8b68 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:28.776759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2daa5a2a-2959-4717-9214-ef02fc4c8b68 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:28.788894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 546/1098] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:28.794533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65ed305-2950-4e66-b9f6-cd3c5e2f897d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:28.879697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:28.894617 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65ed305-2950-4e66-b9f6-cd3c5e2f897d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:28.928728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ee7022-a41f-4a05-ba4a-4d9074eb52e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:28.929348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ee7022-a41f-4a05-ba4a-4d9074eb52e4 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:28.929629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ee7022-a41f-4a05-ba4a-4d9074eb52e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:28.929902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ee7022-a41f-4a05-ba4a-4d9074eb52e4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:28.937334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ee7022-a41f-4a05-ba4a-4d9074eb52e4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:28.938775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ee7022-a41f-4a05-ba4a-4d9074eb52e4 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-66ee7022-a41f-4a05-ba4a-4d9074eb52e4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:24:28.949704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:28.955603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:28.957133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 553/1099] 10.222.0.25 () {56 vars in 1093 bytes} [Mon Sep 26 01:24:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:28.957768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2daa5a2a-2959-4717-9214-ef02fc4c8b68 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:29.057817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2daa5a2a-2959-4717-9214-ef02fc4c8b68 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:29.065323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:29.094571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e8f156-fc42-4b68-97cc-aed21f24aa3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:29.095176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e8f156-fc42-4b68-97cc-aed21f24aa3d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:29.095452 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e8f156-fc42-4b68-97cc-aed21f24aa3d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:29.095723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e8f156-fc42-4b68-97cc-aed21f24aa3d None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:29.096511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e8f156-fc42-4b68-97cc-aed21f24aa3d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:29.096816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e8f156-fc42-4b68-97cc-aed21f24aa3d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:29.098251 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e8f156-fc42-4b68-97cc-aed21f24aa3d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:29.116063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 547/1100] 10.222.0.25 () {56 vars in 1133 bytes} [Mon Sep 26 01:24:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:29.121811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ee7022-a41f-4a05-ba4a-4d9074eb52e4 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:29.219673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ee7022-a41f-4a05-ba4a-4d9074eb52e4 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:29.252018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c6c378-8b5e-4ee8-b9cf-1150e75c464d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:29.252627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c6c378-8b5e-4ee8-b9cf-1150e75c464d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:29.252907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c6c378-8b5e-4ee8-b9cf-1150e75c464d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:29.253178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c6c378-8b5e-4ee8-b9cf-1150e75c464d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:29.267612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:29.320020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c6c378-8b5e-4ee8-b9cf-1150e75c464d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:29.321420 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c6c378-8b5e-4ee8-b9cf-1150e75c464d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:29.418378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 554/1101] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:29.424067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e8f156-fc42-4b68-97cc-aed21f24aa3d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:29.521682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e8f156-fc42-4b68-97cc-aed21f24aa3d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:29.555142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f827ca13-12a6-4fb8-8b55-23425fdb300a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:29.555761 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f827ca13-12a6-4fb8-8b55-23425fdb300a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:29.556039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f827ca13-12a6-4fb8-8b55-23425fdb300a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:29.556313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f827ca13-12a6-4fb8-8b55-23425fdb300a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:29.566071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f827ca13-12a6-4fb8-8b55-23425fdb300a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:29.567511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f827ca13-12a6-4fb8-8b55-23425fdb300a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f827ca13-12a6-4fb8-8b55-23425fdb300a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:29.582008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:29.583100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:24:29.584029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:29.584029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:24:29.584029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:29.584029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 548/1102] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:29.588841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c6c378-8b5e-4ee8-b9cf-1150e75c464d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:29.633913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:29.686425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c6c378-8b5e-4ee8-b9cf-1150e75c464d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:29.720011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f50117-9e55-4fbd-933c-05c66d3c38ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:29.720629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f50117-9e55-4fbd-933c-05c66d3c38ee None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:29.720911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f50117-9e55-4fbd-933c-05c66d3c38ee None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:29.721187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f50117-9e55-4fbd-933c-05c66d3c38ee None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:29.721823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f50117-9e55-4fbd-933c-05c66d3c38ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:29.722129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f50117-9e55-4fbd-933c-05c66d3c38ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:29.723489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f50117-9e55-4fbd-933c-05c66d3c38ee None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:29.734540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 555/1103] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:24:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:24:29.740056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f827ca13-12a6-4fb8-8b55-23425fdb300a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:29.760611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:29.841934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f827ca13-12a6-4fb8-8b55-23425fdb300a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:29.874677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28db813f-ffb1-4284-8223-7d2f2ee45726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:29.875300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28db813f-ffb1-4284-8223-7d2f2ee45726 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:29.875584 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28db813f-ffb1-4284-8223-7d2f2ee45726 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:29.875860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28db813f-ffb1-4284-8223-7d2f2ee45726 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:29.882963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28db813f-ffb1-4284-8223-7d2f2ee45726 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:29.884440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28db813f-ffb1-4284-8223-7d2f2ee45726 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-28db813f-ffb1-4284-8223-7d2f2ee45726 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:24:29.895598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:29.896629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:29.898009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:24:29.898009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:24:29.898009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:24:29.898009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:24:29.898009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:24:29.898009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:24:29.898009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:24:29.898009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:29.900487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 549/1104] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:24:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:24:29.902593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f50117-9e55-4fbd-933c-05c66d3c38ee None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:30.002359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f50117-9e55-4fbd-933c-05c66d3c38ee None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:30.009967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:30.036725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5e1eff-c3c7-4244-8422-a99283c47bd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:30.037333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e1eff-c3c7-4244-8422-a99283c47bd9 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:30.038148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e1eff-c3c7-4244-8422-a99283c47bd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:30.038530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e1eff-c3c7-4244-8422-a99283c47bd9 None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:30.039455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5e1eff-c3c7-4244-8422-a99283c47bd9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:30.039760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5e1eff-c3c7-4244-8422-a99283c47bd9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:30.041209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5e1eff-c3c7-4244-8422-a99283c47bd9 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:30.053202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 556/1105] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:24:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:30.059800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28db813f-ffb1-4284-8223-7d2f2ee45726 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:30.136648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:30.159164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28db813f-ffb1-4284-8223-7d2f2ee45726 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:30.192392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1d40a2-8396-494e-a8a8-fe9bf1701a8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:30.192981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d40a2-8396-494e-a8a8-fe9bf1701a8d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:30.193258 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d40a2-8396-494e-a8a8-fe9bf1701a8d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:30.193528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d40a2-8396-494e-a8a8-fe9bf1701a8d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:30.195048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1d40a2-8396-494e-a8a8-fe9bf1701a8d None admin] RBAC: Adding query filter params (role.id=fe72ae0ebcab44b69a769caf591d6eda, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=381ad7c239f444da90369dc58779b86c) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:30.195361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1d40a2-8396-494e-a8a8-fe9bf1701a8d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:30.196813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1d40a2-8396-494e-a8a8-fe9bf1701a8d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:30.202537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 550/1106] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:24:30 2022] GET /identity/v3/role_assignments?user.id=381ad7c239f444da90369dc58779b86c&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=fe72ae0ebcab44b69a769caf591d6eda => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:30.377466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b19d225-05bd-4752-a610-abb55fb2d81a None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:30.377750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b19d225-05bd-4752-a610-abb55fb2d81a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:30.378025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b19d225-05bd-4752-a610-abb55fb2d81a None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:30.379340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 557/1107] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:30.385782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d32ccb8-df19-4878-8b0a-fc66b4749ae3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:30.386064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d32ccb8-df19-4878-8b0a-fc66b4749ae3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:30.386338 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d32ccb8-df19-4878-8b0a-fc66b4749ae3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:30.405468 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8d32ccb8-df19-4878-8b0a-fc66b4749ae3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:30.406585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8d32ccb8-df19-4878-8b0a-fc66b4749ae3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:30.474147 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 551/1108] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:30.749329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b19d225-05bd-4752-a610-abb55fb2d81a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:30.853722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b19d225-05bd-4752-a610-abb55fb2d81a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:30.888023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c643ce6d-4873-498a-a495-8424b9e0e946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:30.888679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c643ce6d-4873-498a-a495-8424b9e0e946 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:30.888955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c643ce6d-4873-498a-a495-8424b9e0e946 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:30.889226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c643ce6d-4873-498a-a495-8424b9e0e946 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:30.890318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c643ce6d-4873-498a-a495-8424b9e0e946 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:30.891764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c643ce6d-4873-498a-a495-8424b9e0e946 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:30.893341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 558/1109] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:30.899970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d32ccb8-df19-4878-8b0a-fc66b4749ae3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:30.972527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:31.003631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d32ccb8-df19-4878-8b0a-fc66b4749ae3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:31.039573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34f82bb-d187-40a3-82d9-9dbb0e1a0478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:31.040277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34f82bb-d187-40a3-82d9-9dbb0e1a0478 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:31.040638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34f82bb-d187-40a3-82d9-9dbb0e1a0478 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:31.040910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34f82bb-d187-40a3-82d9-9dbb0e1a0478 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:31.041567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34f82bb-d187-40a3-82d9-9dbb0e1a0478 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:31.043016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34f82bb-d187-40a3-82d9-9dbb0e1a0478 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:31.063987 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 552/1110] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:24:30 2022] POST /identity/v3/users => generated 286 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:24:33.244357 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9504c548-486d-4fa8-a6ee-a6c096499713 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:33.244904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9504c548-486d-4fa8-a6ee-a6c096499713 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:33.244904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9504c548-486d-4fa8-a6ee-a6c096499713 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:33.246218 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 559/1111] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:33.251439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b46d78-9c22-42bb-87e8-4cab8f0d93a7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:33.251860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b46d78-9c22-42bb-87e8-4cab8f0d93a7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:33.251980 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b46d78-9c22-42bb-87e8-4cab8f0d93a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:33.269874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e3b46d78-9c22-42bb-87e8-4cab8f0d93a7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:33.271052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e3b46d78-9c22-42bb-87e8-4cab8f0d93a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:33.335373 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 553/1112] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:33.604928 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9504c548-486d-4fa8-a6ee-a6c096499713 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:33.639912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:33.707490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9504c548-486d-4fa8-a6ee-a6c096499713 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:33.740012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e8588c-e2a8-474e-a5c5-af747103639a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:33.741232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e8588c-e2a8-474e-a5c5-af747103639a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:33.741684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e8588c-e2a8-474e-a5c5-af747103639a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:33.741966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e8588c-e2a8-474e-a5c5-af747103639a None admin] PATH_INFO: `/v3/roles/48389faedba2470a885fae19084bb456` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:33.743352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e8588c-e2a8-474e-a5c5-af747103639a None admin] RBAC: Authorizing `identity:get_role(role_id=48389faedba2470a885fae19084bb456)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:33.744737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e8588c-e2a8-474e-a5c5-af747103639a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:33.746019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 560/1113] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:33 2022] GET /identity/v3/roles/48389faedba2470a885fae19084bb456 => generated 223 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:33.751849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b46d78-9c22-42bb-87e8-4cab8f0d93a7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:33.807020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:33.849297 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b46d78-9c22-42bb-87e8-4cab8f0d93a7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:33.883634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9099809-41f1-43a6-9e76-2db3786edc0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:33.884187 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9099809-41f1-43a6-9e76-2db3786edc0f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:33.884499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9099809-41f1-43a6-9e76-2db3786edc0f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:33.884775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9099809-41f1-43a6-9e76-2db3786edc0f None admin] PATH_INFO: `/v3/users/7867ef6b249946309af308a419c840f8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:33.897555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9099809-41f1-43a6-9e76-2db3786edc0f None admin] RBAC: Authorizing `identity:get_user(user_id=7867ef6b249946309af308a419c840f8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:33.898999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9099809-41f1-43a6-9e76-2db3786edc0f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:33.900600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 554/1114] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:33 2022] GET /identity/v3/users/7867ef6b249946309af308a419c840f8 => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:33.906414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e8588c-e2a8-474e-a5c5-af747103639a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:34.003021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e8588c-e2a8-474e-a5c5-af747103639a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:34.004513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:34.035575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4355c6bd-214e-4fd3-8dbd-39f381a5fc8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:34.036174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4355c6bd-214e-4fd3-8dbd-39f381a5fc8e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:34.036481 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4355c6bd-214e-4fd3-8dbd-39f381a5fc8e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:34.036754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4355c6bd-214e-4fd3-8dbd-39f381a5fc8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:34.103468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4355c6bd-214e-4fd3-8dbd-39f381a5fc8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:34.104943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4355c6bd-214e-4fd3-8dbd-39f381a5fc8e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:34.199029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 561/1115] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:34.205008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9099809-41f1-43a6-9e76-2db3786edc0f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:34.301255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9099809-41f1-43a6-9e76-2db3786edc0f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:34.315203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:34.336258 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5e4b2e-a617-491a-8e85-13c995a43e0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:34.336901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5e4b2e-a617-491a-8e85-13c995a43e0a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:34.337183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5e4b2e-a617-491a-8e85-13c995a43e0a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:34.337459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5e4b2e-a617-491a-8e85-13c995a43e0a None admin] PATH_INFO: `/v3/projects/6f6e7af0f4294958b037d48b7b06f959` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:34.338566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5e4b2e-a617-491a-8e85-13c995a43e0a None admin] RBAC: Authorizing `identity:get_project(project_id=6f6e7af0f4294958b037d48b7b06f959)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:34.340004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5e4b2e-a617-491a-8e85-13c995a43e0a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:34.341642 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 555/1116] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:24:34 2022] GET /identity/v3/projects/6f6e7af0f4294958b037d48b7b06f959 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:34.347765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4355c6bd-214e-4fd3-8dbd-39f381a5fc8e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:34.350510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:34.444495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4355c6bd-214e-4fd3-8dbd-39f381a5fc8e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:34.478882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38af147-f9c4-4033-9725-759c011af746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:34.479463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38af147-f9c4-4033-9725-759c011af746 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:34.479742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38af147-f9c4-4033-9725-759c011af746 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:34.480009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38af147-f9c4-4033-9725-759c011af746 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:34.480746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38af147-f9c4-4033-9725-759c011af746 None admin] RBAC: Adding query filter params (role.id=48389faedba2470a885fae19084bb456, scope.project.id=6f6e7af0f4294958b037d48b7b06f959, user.id=7867ef6b249946309af308a419c840f8) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:34.481050 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38af147-f9c4-4033-9725-759c011af746 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:34.482445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38af147-f9c4-4033-9725-759c011af746 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:34.487556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 562/1117] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:24:34 2022] GET /identity/v3/role_assignments?user.id=7867ef6b249946309af308a419c840f8&scope.project.id=6f6e7af0f4294958b037d48b7b06f959&role.id=48389faedba2470a885fae19084bb456 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:36.614379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b55e8e-6928-452a-ac90-8ce7e8a802da None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:36.614939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b55e8e-6928-452a-ac90-8ce7e8a802da None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:36.614939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b55e8e-6928-452a-ac90-8ce7e8a802da None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:36.616852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 556/1118] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:36.622144 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d53d09f-f816-4842-a025-efb6b70f8ba0 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:36.622550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d53d09f-f816-4842-a025-efb6b70f8ba0 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:36.622690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d53d09f-f816-4842-a025-efb6b70f8ba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:36.641087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-4d53d09f-f816-4842-a025-efb6b70f8ba0 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:36.642276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4d53d09f-f816-4842-a025-efb6b70f8ba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:36.700061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:36.707846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 563/1119] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:36.978351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b55e8e-6928-452a-ac90-8ce7e8a802da None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:37.077840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b55e8e-6928-452a-ac90-8ce7e8a802da None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:37.110987 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6352d69-8a29-412b-a65e-dd16316d9a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:37.111604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6352d69-8a29-412b-a65e-dd16316d9a8e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:37.111883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6352d69-8a29-412b-a65e-dd16316d9a8e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:37.112155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6352d69-8a29-412b-a65e-dd16316d9a8e None admin] PATH_INFO: `/v3/roles/48389faedba2470a885fae19084bb456` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:37.113547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6352d69-8a29-412b-a65e-dd16316d9a8e None admin] RBAC: Authorizing `identity:get_role(role_id=48389faedba2470a885fae19084bb456)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:37.114912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6352d69-8a29-412b-a65e-dd16316d9a8e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:37.116829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 557/1120] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:36 2022] GET /identity/v3/roles/48389faedba2470a885fae19084bb456 => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:37.122765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d53d09f-f816-4842-a025-efb6b70f8ba0 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:37.219004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d53d09f-f816-4842-a025-efb6b70f8ba0 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:37.251761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87c11f0-9117-44de-b61b-a5b7e661438d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:37.252313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87c11f0-9117-44de-b61b-a5b7e661438d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:37.252623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87c11f0-9117-44de-b61b-a5b7e661438d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:37.252894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87c11f0-9117-44de-b61b-a5b7e661438d None admin] PATH_INFO: `/v3/users/7867ef6b249946309af308a419c840f8` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:37.254001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87c11f0-9117-44de-b61b-a5b7e661438d None admin] RBAC: Authorizing `identity:get_user(user_id=7867ef6b249946309af308a419c840f8)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:37.255417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87c11f0-9117-44de-b61b-a5b7e661438d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:37.257119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 564/1121] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:37 2022] GET /identity/v3/users/7867ef6b249946309af308a419c840f8 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:37.262788 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6352d69-8a29-412b-a65e-dd16316d9a8e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:37.311913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:37.361376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6352d69-8a29-412b-a65e-dd16316d9a8e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:37.393701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de148fd3-8c8c-472a-b3eb-ceee091b9d2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:37.394311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de148fd3-8c8c-472a-b3eb-ceee091b9d2b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:37.394594 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de148fd3-8c8c-472a-b3eb-ceee091b9d2b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:37.394869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de148fd3-8c8c-472a-b3eb-ceee091b9d2b None admin] PATH_INFO: `/v3/projects/6f6e7af0f4294958b037d48b7b06f959` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:37.395979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de148fd3-8c8c-472a-b3eb-ceee091b9d2b None admin] RBAC: Authorizing `identity:get_project(project_id=6f6e7af0f4294958b037d48b7b06f959)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:37.397973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de148fd3-8c8c-472a-b3eb-ceee091b9d2b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:37.399582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 558/1122] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:24:37 2022] GET /identity/v3/projects/6f6e7af0f4294958b037d48b7b06f959 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:37.405573 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87c11f0-9117-44de-b61b-a5b7e661438d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:37.465941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:37.503321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87c11f0-9117-44de-b61b-a5b7e661438d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:37.536305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87177d5f-bfdd-400a-815e-9b1531cfa4b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:37.536974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87177d5f-bfdd-400a-815e-9b1531cfa4b5 None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:37.537290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87177d5f-bfdd-400a-815e-9b1531cfa4b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:37.537711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87177d5f-bfdd-400a-815e-9b1531cfa4b5 None admin] PATH_INFO: `/v3/projects/6f6e7af0f4294958b037d48b7b06f959/users/7867ef6b249946309af308a419c840f8/roles/48389faedba2470a885fae19084bb456` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:37.539852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87177d5f-bfdd-400a-815e-9b1531cfa4b5 None admin] RBAC: Authorizing `identity:create_grant(project_id=6f6e7af0f4294958b037d48b7b06f959, user_id=7867ef6b249946309af308a419c840f8, role_id=48389faedba2470a885fae19084bb456)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:37.541301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87177d5f-bfdd-400a-815e-9b1531cfa4b5 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:37.554118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 565/1123] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:24:37 2022] PUT /identity/v3/projects/6f6e7af0f4294958b037d48b7b06f959/users/7867ef6b249946309af308a419c840f8/roles/48389faedba2470a885fae19084bb456 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:24:39.677559 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0d2dd-27e0-457d-9efc-da2b769e36ad None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:39.678160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0d2dd-27e0-457d-9efc-da2b769e36ad None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:39.678160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0d2dd-27e0-457d-9efc-da2b769e36ad None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:39.679420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 559/1124] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:39.684592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32faa5b3-92a7-455e-afec-5fd66a81a6a1 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:39.685011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32faa5b3-92a7-455e-afec-5fd66a81a6a1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:39.685154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32faa5b3-92a7-455e-afec-5fd66a81a6a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:39.704667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-32faa5b3-92a7-455e-afec-5fd66a81a6a1 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:39.705881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-32faa5b3-92a7-455e-afec-5fd66a81a6a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:39.770635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 566/1125] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:40.040088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e0d2dd-27e0-457d-9efc-da2b769e36ad None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:40.139794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e0d2dd-27e0-457d-9efc-da2b769e36ad None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:40.157396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:40.174775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a249c8-cbb6-4c88-8214-13b225fd23da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:40.175387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a249c8-cbb6-4c88-8214-13b225fd23da None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:40.175664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a249c8-cbb6-4c88-8214-13b225fd23da None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:40.175934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a249c8-cbb6-4c88-8214-13b225fd23da None admin] PATH_INFO: `/v3/roles/48389faedba2470a885fae19084bb456` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:40.177311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a249c8-cbb6-4c88-8214-13b225fd23da None admin] RBAC: Authorizing `identity:get_role(role_id=48389faedba2470a885fae19084bb456)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:40.178978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a249c8-cbb6-4c88-8214-13b225fd23da None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:40.180280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 560/1126] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:40 2022] GET /identity/v3/roles/48389faedba2470a885fae19084bb456 => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:40.186283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32faa5b3-92a7-455e-afec-5fd66a81a6a1 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:40.239558 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:40.282895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32faa5b3-92a7-455e-afec-5fd66a81a6a1 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:40.316494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e929c0fe-bc31-4242-824f-d578931f7231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:40.317052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e929c0fe-bc31-4242-824f-d578931f7231 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:40.317331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e929c0fe-bc31-4242-824f-d578931f7231 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:40.317603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e929c0fe-bc31-4242-824f-d578931f7231 None admin] PATH_INFO: `/v3/users/7867ef6b249946309af308a419c840f8` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:40.319481 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e929c0fe-bc31-4242-824f-d578931f7231 None admin] RBAC: Authorizing `identity:get_user(user_id=7867ef6b249946309af308a419c840f8)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:40.320936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e929c0fe-bc31-4242-824f-d578931f7231 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:40.322460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 567/1127] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:40 2022] GET /identity/v3/users/7867ef6b249946309af308a419c840f8 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:40.328150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a249c8-cbb6-4c88-8214-13b225fd23da None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:40.424304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a249c8-cbb6-4c88-8214-13b225fd23da None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:40.458812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4759b234-1690-4ff6-859f-3434d4df32fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:40.459400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4759b234-1690-4ff6-859f-3434d4df32fd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:40.459682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4759b234-1690-4ff6-859f-3434d4df32fd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:40.459958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4759b234-1690-4ff6-859f-3434d4df32fd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:40.513952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:40.527298 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4759b234-1690-4ff6-859f-3434d4df32fd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:40.528782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4759b234-1690-4ff6-859f-3434d4df32fd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:40.623668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 561/1128] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:40.629506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e929c0fe-bc31-4242-824f-d578931f7231 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:40.725086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e929c0fe-bc31-4242-824f-d578931f7231 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:40.751115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:40.762720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2315a23-c567-4018-a67b-75f83f4ac690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:40.763338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2315a23-c567-4018-a67b-75f83f4ac690 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:40.763615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2315a23-c567-4018-a67b-75f83f4ac690 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:40.763885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2315a23-c567-4018-a67b-75f83f4ac690 None admin] PATH_INFO: `/v3/projects/6f6e7af0f4294958b037d48b7b06f959` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:40.764997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2315a23-c567-4018-a67b-75f83f4ac690 None admin] RBAC: Authorizing `identity:get_project(project_id=6f6e7af0f4294958b037d48b7b06f959)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:40.766417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2315a23-c567-4018-a67b-75f83f4ac690 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:40.768011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 568/1129] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:24:40 2022] GET /identity/v3/projects/6f6e7af0f4294958b037d48b7b06f959 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:40.774233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4759b234-1690-4ff6-859f-3434d4df32fd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:40.855447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:40.870531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4759b234-1690-4ff6-859f-3434d4df32fd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:40.903341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee5b86c-668f-4513-8b0f-ceb4c0b1730d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:40.903866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee5b86c-668f-4513-8b0f-ceb4c0b1730d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:40.904140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee5b86c-668f-4513-8b0f-ceb4c0b1730d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:40.904438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee5b86c-668f-4513-8b0f-ceb4c0b1730d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:40.905144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee5b86c-668f-4513-8b0f-ceb4c0b1730d None admin] RBAC: Adding query filter params (role.id=48389faedba2470a885fae19084bb456, scope.project.id=6f6e7af0f4294958b037d48b7b06f959, user.id=7867ef6b249946309af308a419c840f8) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:40.905447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee5b86c-668f-4513-8b0f-ceb4c0b1730d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:40.906868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee5b86c-668f-4513-8b0f-ceb4c0b1730d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:40.912570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 562/1130] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:24:40 2022] GET /identity/v3/role_assignments?user.id=7867ef6b249946309af308a419c840f8&scope.project.id=6f6e7af0f4294958b037d48b7b06f959&role.id=48389faedba2470a885fae19084bb456 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:43.068451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39eaa6e-8cfb-4b83-b3e0-b450b59cfae5 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:43.068897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39eaa6e-8cfb-4b83-b3e0-b450b59cfae5 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:43.068989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39eaa6e-8cfb-4b83-b3e0-b450b59cfae5 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:43.070310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 569/1131] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:43.075941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbf7f3-3f3c-409f-824d-9d6c459022f3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:43.076364 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbf7f3-3f3c-409f-824d-9d6c459022f3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:43.076522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbf7f3-3f3c-409f-824d-9d6c459022f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:43.094510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-78dbf7f3-3f3c-409f-824d-9d6c459022f3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:43.095700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-78dbf7f3-3f3c-409f-824d-9d6c459022f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:43.160974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 563/1132] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:43.426019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39eaa6e-8cfb-4b83-b3e0-b450b59cfae5 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:43.525483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39eaa6e-8cfb-4b83-b3e0-b450b59cfae5 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:43.557192 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c333683-3b06-45fd-b267-3bd24b45cc37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:43.557805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c333683-3b06-45fd-b267-3bd24b45cc37 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:43.558080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c333683-3b06-45fd-b267-3bd24b45cc37 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:43.558348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c333683-3b06-45fd-b267-3bd24b45cc37 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:43.559430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c333683-3b06-45fd-b267-3bd24b45cc37 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:43.560871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c333683-3b06-45fd-b267-3bd24b45cc37 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:43.562380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 570/1133] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:43.568109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dbf7f3-3f3c-409f-824d-9d6c459022f3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:43.646064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:43.665912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dbf7f3-3f3c-409f-824d-9d6c459022f3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:43.699509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb44bb5-fd90-4edf-a9f9-987a9b06358c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:43.700038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb44bb5-fd90-4edf-a9f9-987a9b06358c None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:43.700313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb44bb5-fd90-4edf-a9f9-987a9b06358c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:43.700613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb44bb5-fd90-4edf-a9f9-987a9b06358c None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:43.701264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb44bb5-fd90-4edf-a9f9-987a9b06358c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:43.702682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb44bb5-fd90-4edf-a9f9-987a9b06358c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:43.731027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 564/1134] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:24:43 2022] POST /identity/v3/projects => generated 311 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:24:45.915034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1f775-7af1-4553-929d-02c1d6a17441 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:45.915614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1f775-7af1-4553-929d-02c1d6a17441 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:45.915614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1f775-7af1-4553-929d-02c1d6a17441 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:45.916941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 571/1135] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:45.922184 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bd4071-d9ba-4267-87f0-5cd57e726d7c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:45.922603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bd4071-d9ba-4267-87f0-5cd57e726d7c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:45.922732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bd4071-d9ba-4267-87f0-5cd57e726d7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:45.941739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f5bd4071-d9ba-4267-87f0-5cd57e726d7c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:45.942944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f5bd4071-d9ba-4267-87f0-5cd57e726d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:46.008270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 565/1136] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:46.278110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1f775-7af1-4553-929d-02c1d6a17441 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:46.336298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:46.377024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1f775-7af1-4553-929d-02c1d6a17441 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:46.409115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed27a29-cc99-4f42-8193-49dfd23bf96a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:46.409721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed27a29-cc99-4f42-8193-49dfd23bf96a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:46.410000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed27a29-cc99-4f42-8193-49dfd23bf96a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:46.410269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed27a29-cc99-4f42-8193-49dfd23bf96a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:46.411357 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed27a29-cc99-4f42-8193-49dfd23bf96a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:46.412797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed27a29-cc99-4f42-8193-49dfd23bf96a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:46.414330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 572/1137] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:24:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:46.419972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bd4071-d9ba-4267-87f0-5cd57e726d7c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:46.480765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:46.519545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bd4071-d9ba-4267-87f0-5cd57e726d7c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:46.552907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d83417-21fd-422b-a5dd-4f1887be2d6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:46.553430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d83417-21fd-422b-a5dd-4f1887be2d6a None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:46.553704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d83417-21fd-422b-a5dd-4f1887be2d6a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:46.554322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d83417-21fd-422b-a5dd-4f1887be2d6a None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:46.554917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d83417-21fd-422b-a5dd-4f1887be2d6a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:46.556335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d83417-21fd-422b-a5dd-4f1887be2d6a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:46.577176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 566/1138] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:24:46 2022] POST /identity/v3/users => generated 286 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:24:48.739860 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc9a955-e8e5-4ed5-9134-ae3b7d3362e4 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:48.740426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc9a955-e8e5-4ed5-9134-ae3b7d3362e4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:48.740426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc9a955-e8e5-4ed5-9134-ae3b7d3362e4 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:48.741896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 573/1139] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:48.747083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee1e2d-cf8c-4237-88e3-b10da83b1cbc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:48.747521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee1e2d-cf8c-4237-88e3-b10da83b1cbc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:48.747638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee1e2d-cf8c-4237-88e3-b10da83b1cbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:48.765994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-22ee1e2d-cf8c-4237-88e3-b10da83b1cbc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:48.767193 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-22ee1e2d-cf8c-4237-88e3-b10da83b1cbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:48.833101 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 567/1140] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:49.102829 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc9a955-e8e5-4ed5-9134-ae3b7d3362e4 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:49.199813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc9a955-e8e5-4ed5-9134-ae3b7d3362e4 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:49.224708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:49.233477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5926b781-af72-4e57-a7bf-13902d8608fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:49.234086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5926b781-af72-4e57-a7bf-13902d8608fa None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:49.234361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5926b781-af72-4e57-a7bf-13902d8608fa None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:49.234628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5926b781-af72-4e57-a7bf-13902d8608fa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:49.244848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5926b781-af72-4e57-a7bf-13902d8608fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-5926b781-af72-4e57-a7bf-13902d8608fa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:49.249537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:49.250452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:49.251475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 574/1141] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:24:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:49.256534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee1e2d-cf8c-4237-88e3-b10da83b1cbc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:49.304311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:49.353866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee1e2d-cf8c-4237-88e3-b10da83b1cbc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:49.387186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6d3b6d-530d-481a-9b2e-a12c12833e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:49.387777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6d3b6d-530d-481a-9b2e-a12c12833e8f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:49.388055 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6d3b6d-530d-481a-9b2e-a12c12833e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:49.388325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6d3b6d-530d-481a-9b2e-a12c12833e8f None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:49.389007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6d3b6d-530d-481a-9b2e-a12c12833e8f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:49.389310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6d3b6d-530d-481a-9b2e-a12c12833e8f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:49.390678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6d3b6d-530d-481a-9b2e-a12c12833e8f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:49.398502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 568/1142] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:24:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:49.404437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5926b781-af72-4e57-a7bf-13902d8608fa None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:49.501151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5926b781-af72-4e57-a7bf-13902d8608fa None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:49.533703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e3d73-29f5-4f4c-aa84-585d3ecf795d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:49.534304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e3d73-29f5-4f4c-aa84-585d3ecf795d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:49.534584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e3d73-29f5-4f4c-aa84-585d3ecf795d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:49.534857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e3d73-29f5-4f4c-aa84-585d3ecf795d None admin] PATH_INFO: `/v3/users/03feb1b9ebfd4ce890d370e467c10584` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:49.548847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e3d73-29f5-4f4c-aa84-585d3ecf795d None admin] RBAC: Authorizing `identity:get_user(user_id=03feb1b9ebfd4ce890d370e467c10584)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:49.550282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e3d73-29f5-4f4c-aa84-585d3ecf795d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:49.551830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 575/1143] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:49 2022] GET /identity/v3/users/03feb1b9ebfd4ce890d370e467c10584 => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:49.558335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6d3b6d-530d-481a-9b2e-a12c12833e8f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:49.655583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6d3b6d-530d-481a-9b2e-a12c12833e8f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:49.666081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:49.689652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32631e2-253a-4d99-b5fb-b2558c2938d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:49.690242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32631e2-253a-4d99-b5fb-b2558c2938d6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:49.690520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32631e2-253a-4d99-b5fb-b2558c2938d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:49.690790 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32631e2-253a-4d99-b5fb-b2558c2938d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:49.756208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32631e2-253a-4d99-b5fb-b2558c2938d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:49.757720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32631e2-253a-4d99-b5fb-b2558c2938d6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:49.854561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 569/1144] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:49.860546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e3d73-29f5-4f4c-aa84-585d3ecf795d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:49.905274 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:49.956521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e3d73-29f5-4f4c-aa84-585d3ecf795d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:49.989634 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b783d526-9d79-4738-a44f-315fd0f8e8fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:49.990241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b783d526-9d79-4738-a44f-315fd0f8e8fe None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:49.990519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b783d526-9d79-4738-a44f-315fd0f8e8fe None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:49.990790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b783d526-9d79-4738-a44f-315fd0f8e8fe None admin] PATH_INFO: `/v3/projects/afb52b0c373a41f1ad061edf3c9c77f4` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:49.992672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b783d526-9d79-4738-a44f-315fd0f8e8fe None admin] RBAC: Authorizing `identity:get_project(project_id=afb52b0c373a41f1ad061edf3c9c77f4)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:49.994089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b783d526-9d79-4738-a44f-315fd0f8e8fe None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:49.995666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 576/1145] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:24:49 2022] GET /identity/v3/projects/afb52b0c373a41f1ad061edf3c9c77f4 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:50.001683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32631e2-253a-4d99-b5fb-b2558c2938d6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:50.013850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:50.099614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32631e2-253a-4d99-b5fb-b2558c2938d6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:50.133748 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75e2060-ca45-4167-b9f6-cad9689c0d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:50.134273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75e2060-ca45-4167-b9f6-cad9689c0d73 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:50.134549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75e2060-ca45-4167-b9f6-cad9689c0d73 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:50.134816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75e2060-ca45-4167-b9f6-cad9689c0d73 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:50.135522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75e2060-ca45-4167-b9f6-cad9689c0d73 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=afb52b0c373a41f1ad061edf3c9c77f4, user.id=03feb1b9ebfd4ce890d370e467c10584) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:50.135823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75e2060-ca45-4167-b9f6-cad9689c0d73 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:50.137268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75e2060-ca45-4167-b9f6-cad9689c0d73 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:50.142442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 570/1146] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:24:50 2022] GET /identity/v3/role_assignments?user.id=03feb1b9ebfd4ce890d370e467c10584&scope.project.id=afb52b0c373a41f1ad061edf3c9c77f4&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:52.267189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4673d7-83c0-4096-8eb7-9c8dcb6451d5 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:52.267711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4673d7-83c0-4096-8eb7-9c8dcb6451d5 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:52.267821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4673d7-83c0-4096-8eb7-9c8dcb6451d5 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:52.269091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 577/1147] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:52.274863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6deffb-1f19-4421-9c27-dacbf2225eaf None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:52.275286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6deffb-1f19-4421-9c27-dacbf2225eaf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:52.275397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6deffb-1f19-4421-9c27-dacbf2225eaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:52.294920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ff6deffb-1f19-4421-9c27-dacbf2225eaf None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:52.296092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ff6deffb-1f19-4421-9c27-dacbf2225eaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:52.361558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 571/1148] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:52.630538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4673d7-83c0-4096-8eb7-9c8dcb6451d5 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:52.729491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4673d7-83c0-4096-8eb7-9c8dcb6451d5 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:52.742487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:52.762892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70eef78e-659b-4a9c-aeb8-7587722864a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:52.763499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70eef78e-659b-4a9c-aeb8-7587722864a8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:52.763773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70eef78e-659b-4a9c-aeb8-7587722864a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:52.764043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70eef78e-659b-4a9c-aeb8-7587722864a8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:52.774460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70eef78e-659b-4a9c-aeb8-7587722864a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-70eef78e-659b-4a9c-aeb8-7587722864a8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:52.779187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:52.780287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:52.781562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 578/1149] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:24:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:52.786299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6deffb-1f19-4421-9c27-dacbf2225eaf None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:52.789988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:52.886541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6deffb-1f19-4421-9c27-dacbf2225eaf None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:52.919841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bc8d97-abaf-4f33-bc66-72ea91f4fa64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:52.920454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc8d97-abaf-4f33-bc66-72ea91f4fa64 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:52.920734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc8d97-abaf-4f33-bc66-72ea91f4fa64 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:52.921005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc8d97-abaf-4f33-bc66-72ea91f4fa64 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:52.921657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bc8d97-abaf-4f33-bc66-72ea91f4fa64 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:52.921958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bc8d97-abaf-4f33-bc66-72ea91f4fa64 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:52.923321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bc8d97-abaf-4f33-bc66-72ea91f4fa64 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:52.931248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 572/1150] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:24:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:52.937080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70eef78e-659b-4a9c-aeb8-7587722864a8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:53.035067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70eef78e-659b-4a9c-aeb8-7587722864a8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:53.068167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42621ce0-7a8c-415d-b474-2db9501cdbde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:53.068821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42621ce0-7a8c-415d-b474-2db9501cdbde None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:53.069110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42621ce0-7a8c-415d-b474-2db9501cdbde None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:53.069970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42621ce0-7a8c-415d-b474-2db9501cdbde None admin] PATH_INFO: `/v3/users/03feb1b9ebfd4ce890d370e467c10584` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:53.071081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42621ce0-7a8c-415d-b474-2db9501cdbde None admin] RBAC: Authorizing `identity:get_user(user_id=03feb1b9ebfd4ce890d370e467c10584)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:53.072543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42621ce0-7a8c-415d-b474-2db9501cdbde None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:53.074089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 579/1151] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:52 2022] GET /identity/v3/users/03feb1b9ebfd4ce890d370e467c10584 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:53.079808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bc8d97-abaf-4f33-bc66-72ea91f4fa64 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:53.143991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:53.177466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bc8d97-abaf-4f33-bc66-72ea91f4fa64 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:53.210706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315d7c7-cea4-4c11-bf7b-ccdba9730e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:53.211309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315d7c7-cea4-4c11-bf7b-ccdba9730e59 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:53.211589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315d7c7-cea4-4c11-bf7b-ccdba9730e59 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:53.211863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315d7c7-cea4-4c11-bf7b-ccdba9730e59 None admin] PATH_INFO: `/v3/projects/afb52b0c373a41f1ad061edf3c9c77f4` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:53.212980 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2315d7c7-cea4-4c11-bf7b-ccdba9730e59 None admin] RBAC: Authorizing `identity:get_project(project_id=afb52b0c373a41f1ad061edf3c9c77f4)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:53.214401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2315d7c7-cea4-4c11-bf7b-ccdba9730e59 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:53.215998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 573/1152] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:24:53 2022] GET /identity/v3/projects/afb52b0c373a41f1ad061edf3c9c77f4 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:53.221932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42621ce0-7a8c-415d-b474-2db9501cdbde None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:53.263069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:53.319640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42621ce0-7a8c-415d-b474-2db9501cdbde None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:53.352569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b122fc-e5e0-4739-9252-312a27f3ea7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:53.353213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b122fc-e5e0-4739-9252-312a27f3ea7d None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:53.353490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b122fc-e5e0-4739-9252-312a27f3ea7d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:53.353761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b122fc-e5e0-4739-9252-312a27f3ea7d None admin] PATH_INFO: `/v3/projects/afb52b0c373a41f1ad061edf3c9c77f4/users/03feb1b9ebfd4ce890d370e467c10584/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:53.355629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b122fc-e5e0-4739-9252-312a27f3ea7d None admin] RBAC: Authorizing `identity:create_grant(project_id=afb52b0c373a41f1ad061edf3c9c77f4, user_id=03feb1b9ebfd4ce890d370e467c10584, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:53.357068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b122fc-e5e0-4739-9252-312a27f3ea7d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:53.370576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 580/1153] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:24:53 2022] PUT /identity/v3/projects/afb52b0c373a41f1ad061edf3c9c77f4/users/03feb1b9ebfd4ce890d370e467c10584/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:24:55.471298 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13873b-88a4-41b9-b776-4786446798b2 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:55.471860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13873b-88a4-41b9-b776-4786446798b2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:55.471860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13873b-88a4-41b9-b776-4786446798b2 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:55.473244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 574/1154] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:55.478904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f737d-4d62-4510-af6e-f2104bf8516e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:55.479305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f737d-4d62-4510-af6e-f2104bf8516e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:55.479451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f737d-4d62-4510-af6e-f2104bf8516e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:55.497190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-2b1f737d-4d62-4510-af6e-f2104bf8516e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:55.498361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2b1f737d-4d62-4510-af6e-f2104bf8516e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:55.562808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 581/1155] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:55.832171 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba13873b-88a4-41b9-b776-4786446798b2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:55.932913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba13873b-88a4-41b9-b776-4786446798b2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:55.965717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a567c4b0-e7b0-4e73-9b05-b810867ca84e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:55.966324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a567c4b0-e7b0-4e73-9b05-b810867ca84e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:55.966605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a567c4b0-e7b0-4e73-9b05-b810867ca84e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:55.966872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a567c4b0-e7b0-4e73-9b05-b810867ca84e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:55.968658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:55.979128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a567c4b0-e7b0-4e73-9b05-b810867ca84e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a567c4b0-e7b0-4e73-9b05-b810867ca84e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:24:55.983873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:55.985102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:24:55.986244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 575/1156] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:24:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:55.991021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1f737d-4d62-4510-af6e-f2104bf8516e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:56.019548 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:56.090147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1f737d-4d62-4510-af6e-f2104bf8516e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:56.122448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c3830d-4624-4bbb-b613-ccbd32af87b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:56.123041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c3830d-4624-4bbb-b613-ccbd32af87b2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:56.123318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c3830d-4624-4bbb-b613-ccbd32af87b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:56.123592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c3830d-4624-4bbb-b613-ccbd32af87b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:56.124241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c3830d-4624-4bbb-b613-ccbd32af87b2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:56.124564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c3830d-4624-4bbb-b613-ccbd32af87b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:56.125907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c3830d-4624-4bbb-b613-ccbd32af87b2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:56.133634 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 582/1157] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:24:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:56.139491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a567c4b0-e7b0-4e73-9b05-b810867ca84e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:56.235885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a567c4b0-e7b0-4e73-9b05-b810867ca84e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:56.267990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8729752f-8d1d-4cdd-8d23-48b192cd48b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:56.268618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8729752f-8d1d-4cdd-8d23-48b192cd48b3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:56.268894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8729752f-8d1d-4cdd-8d23-48b192cd48b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:56.269160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8729752f-8d1d-4cdd-8d23-48b192cd48b3 None admin] PATH_INFO: `/v3/users/03feb1b9ebfd4ce890d370e467c10584` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:56.270253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8729752f-8d1d-4cdd-8d23-48b192cd48b3 None admin] RBAC: Authorizing `identity:get_user(user_id=03feb1b9ebfd4ce890d370e467c10584)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:56.271664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8729752f-8d1d-4cdd-8d23-48b192cd48b3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:56.273838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 576/1158] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:24:56 2022] GET /identity/v3/users/03feb1b9ebfd4ce890d370e467c10584 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:56.279642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c3830d-4624-4bbb-b613-ccbd32af87b2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:56.367292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:56.375671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c3830d-4624-4bbb-b613-ccbd32af87b2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:56.408605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88274510-9c02-4283-aa74-19334ed80b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:56.409191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88274510-9c02-4283-aa74-19334ed80b6f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:56.409466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88274510-9c02-4283-aa74-19334ed80b6f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:56.409736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88274510-9c02-4283-aa74-19334ed80b6f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:56.474241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88274510-9c02-4283-aa74-19334ed80b6f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:56.476014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88274510-9c02-4283-aa74-19334ed80b6f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:56.560645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:56.570881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 583/1159] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:56 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:56.576873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8729752f-8d1d-4cdd-8d23-48b192cd48b3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:56.639056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:56.677410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8729752f-8d1d-4cdd-8d23-48b192cd48b3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:56.711013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eb1600-e00f-44c1-b767-1cd5971386cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:56.711608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eb1600-e00f-44c1-b767-1cd5971386cc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:56.711884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eb1600-e00f-44c1-b767-1cd5971386cc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:56.712155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eb1600-e00f-44c1-b767-1cd5971386cc None admin] PATH_INFO: `/v3/projects/afb52b0c373a41f1ad061edf3c9c77f4` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:56.713273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3eb1600-e00f-44c1-b767-1cd5971386cc None admin] RBAC: Authorizing `identity:get_project(project_id=afb52b0c373a41f1ad061edf3c9c77f4)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:56.714692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3eb1600-e00f-44c1-b767-1cd5971386cc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:56.716256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 577/1160] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:24:56 2022] GET /identity/v3/projects/afb52b0c373a41f1ad061edf3c9c77f4 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:56.722402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88274510-9c02-4283-aa74-19334ed80b6f None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:56.819705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88274510-9c02-4283-aa74-19334ed80b6f None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:56.853270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad66bfa-b099-4ad3-a391-e07bbc034353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:56.853851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad66bfa-b099-4ad3-a391-e07bbc034353 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:56.854126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad66bfa-b099-4ad3-a391-e07bbc034353 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:56.854389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad66bfa-b099-4ad3-a391-e07bbc034353 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:56.855098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad66bfa-b099-4ad3-a391-e07bbc034353 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=afb52b0c373a41f1ad061edf3c9c77f4, user.id=03feb1b9ebfd4ce890d370e467c10584) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:56.855401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad66bfa-b099-4ad3-a391-e07bbc034353 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:56.856825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad66bfa-b099-4ad3-a391-e07bbc034353 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:56.862323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 584/1161] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:24:56 2022] GET /identity/v3/role_assignments?user.id=03feb1b9ebfd4ce890d370e467c10584&scope.project.id=afb52b0c373a41f1ad061edf3c9c77f4&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:24:59.015644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfab880-ad23-4579-b134-01bfef299972 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:59.016195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfab880-ad23-4579-b134-01bfef299972 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:59.016195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfab880-ad23-4579-b134-01bfef299972 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:59.017554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 578/1162] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:24:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:24:59.023420 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3adfe34-c300-498e-807d-34af895acb2e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:59.023855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3adfe34-c300-498e-807d-34af895acb2e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:59.023951 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3adfe34-c300-498e-807d-34af895acb2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:59.042883 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-d3adfe34-c300-498e-807d-34af895acb2e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:24:59.044070 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d3adfe34-c300-498e-807d-34af895acb2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:24:59.093620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:59.110578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 585/1163] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:24:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:24:59.378706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfab880-ad23-4579-b134-01bfef299972 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:59.478400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfab880-ad23-4579-b134-01bfef299972 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:59.505010 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:59.511651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35bdaa4-b5ed-4134-97d6-286388c2dfd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:59.512245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35bdaa4-b5ed-4134-97d6-286388c2dfd3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:59.512553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35bdaa4-b5ed-4134-97d6-286388c2dfd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:59.512829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35bdaa4-b5ed-4134-97d6-286388c2dfd3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:59.577948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35bdaa4-b5ed-4134-97d6-286388c2dfd3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:59.579389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35bdaa4-b5ed-4134-97d6-286388c2dfd3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:59.674803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 579/1164] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:24:59 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:24:59.680724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3adfe34-c300-498e-807d-34af895acb2e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:59.776133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3adfe34-c300-498e-807d-34af895acb2e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:59.808934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17eec3d9-6ece-4a6b-8b83-7e479aa22724 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:59.809538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eec3d9-6ece-4a6b-8b83-7e479aa22724 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:59.809814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eec3d9-6ece-4a6b-8b83-7e479aa22724 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:59.810083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eec3d9-6ece-4a6b-8b83-7e479aa22724 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:59.816418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17eec3d9-6ece-4a6b-8b83-7e479aa22724 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:59.817830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17eec3d9-6ece-4a6b-8b83-7e479aa22724 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-17eec3d9-6ece-4a6b-8b83-7e479aa22724 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:24:59.828941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:24:59.830027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:24:59.831033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 586/1165] 10.222.0.25 () {56 vars in 1117 bytes} [Mon Sep 26 01:24:59 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:24:59.836855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35bdaa4-b5ed-4134-97d6-286388c2dfd3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:24:59.866901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:24:59.934850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35bdaa4-b5ed-4134-97d6-286388c2dfd3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:24:59.968220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae039b-97a4-49c9-9283-53e26dcf02d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:24:59.968836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae039b-97a4-49c9-9283-53e26dcf02d4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:24:59.969112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae039b-97a4-49c9-9283-53e26dcf02d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:24:59.969379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae039b-97a4-49c9-9283-53e26dcf02d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:24:59.970002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae039b-97a4-49c9-9283-53e26dcf02d4 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:24:59.970301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae039b-97a4-49c9-9283-53e26dcf02d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:24:59.971656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae039b-97a4-49c9-9283-53e26dcf02d4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:24:59.979464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 580/1166] 10.222.0.25 () {56 vars in 1115 bytes} [Mon Sep 26 01:24:59 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:02.111690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e3ec28-2069-4904-8d9d-50e18585079d None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:02.112172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e3ec28-2069-4904-8d9d-50e18585079d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:02.112282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e3ec28-2069-4904-8d9d-50e18585079d None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:02.113571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 587/1167] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:25:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:25:02.119504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8cf6b2-8957-4ef3-9b54-f0da50f2c155 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:02.119928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8cf6b2-8957-4ef3-9b54-f0da50f2c155 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:02.120061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8cf6b2-8957-4ef3-9b54-f0da50f2c155 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:02.138592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8f8cf6b2-8957-4ef3-9b54-f0da50f2c155 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:25:02.139785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8f8cf6b2-8957-4ef3-9b54-f0da50f2c155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:25:02.204778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 581/1168] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:25:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:25:02.473428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e3ec28-2069-4904-8d9d-50e18585079d None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:02.520503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:02.573291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e3ec28-2069-4904-8d9d-50e18585079d None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:02.605902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6939e3a2-8acc-423b-a1d1-1594ad85edc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:02.606492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6939e3a2-8acc-423b-a1d1-1594ad85edc6 None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:02.606775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6939e3a2-8acc-423b-a1d1-1594ad85edc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:02.607050 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6939e3a2-8acc-423b-a1d1-1594ad85edc6 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:02.607715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6939e3a2-8acc-423b-a1d1-1594ad85edc6 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:02.609096 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6939e3a2-8acc-423b-a1d1-1594ad85edc6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:02.624476 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 588/1169] 10.222.0.25 () {60 vars in 1138 bytes} [Mon Sep 26 01:25:02 2022] POST /identity/v3/domains => generated 267 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:25:04.774243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b342231f-5504-4a1c-b0fa-ff0a40a946b9 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:04.774804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b342231f-5504-4a1c-b0fa-ff0a40a946b9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:04.774804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b342231f-5504-4a1c-b0fa-ff0a40a946b9 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:04.776132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 582/1170] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:25:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:25:04.781332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aebf610-f3b1-4cc2-b06a-322bba924fe2 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:04.781714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aebf610-f3b1-4cc2-b06a-322bba924fe2 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:04.781887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aebf610-f3b1-4cc2-b06a-322bba924fe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:04.799584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0aebf610-f3b1-4cc2-b06a-322bba924fe2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:25:04.800814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0aebf610-f3b1-4cc2-b06a-322bba924fe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:25:04.869152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 589/1171] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:25:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 26 01:25:05.135501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b342231f-5504-4a1c-b0fa-ff0a40a946b9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:05.159962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:05.234860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b342231f-5504-4a1c-b0fa-ff0a40a946b9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:05.267546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b80354-6846-43d9-84ae-24506b680ef5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:05.268162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b80354-6846-43d9-84ae-24506b680ef5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:05.268467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b80354-6846-43d9-84ae-24506b680ef5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:05.268743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b80354-6846-43d9-84ae-24506b680ef5 None admin] PATH_INFO: `/v3/domains/d9709e4cb018446cb32b0e57f5198b7b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:05.281517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b80354-6846-43d9-84ae-24506b680ef5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=d9709e4cb018446cb32b0e57f5198b7b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:05.282974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b80354-6846-43d9-84ae-24506b680ef5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:05.286883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 583/1172] 10.222.0.25 () {56 vars in 1183 bytes} [Mon Sep 26 01:25:05 2022] GET /identity/v3/domains/d9709e4cb018446cb32b0e57f5198b7b => generated 267 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:05.293029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aebf610-f3b1-4cc2-b06a-322bba924fe2 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:05.342176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:05.409726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aebf610-f3b1-4cc2-b06a-322bba924fe2 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:05.444458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f1162a-e687-4e7c-95e6-b71db76488dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:05.445073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f1162a-e687-4e7c-95e6-b71db76488dc None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:05.445353 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f1162a-e687-4e7c-95e6-b71db76488dc None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:05.445627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f1162a-e687-4e7c-95e6-b71db76488dc None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:05.446277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f1162a-e687-4e7c-95e6-b71db76488dc None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:05.447698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f1162a-e687-4e7c-95e6-b71db76488dc None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:05.474009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 590/1173] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:25:05 2022] POST /identity/v3/projects => generated 361 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:25:07.652299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742e34ae-e93a-444d-8e6d-4a714c258036 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:07.652882 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742e34ae-e93a-444d-8e6d-4a714c258036 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:07.652882 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742e34ae-e93a-444d-8e6d-4a714c258036 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:07.654205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 584/1174] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:25:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:25:07.659296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beed1b98-6279-4652-8a04-f619d3029a9a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:07.659736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beed1b98-6279-4652-8a04-f619d3029a9a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:07.659837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beed1b98-6279-4652-8a04-f619d3029a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:07.678674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-beed1b98-6279-4652-8a04-f619d3029a9a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:25:07.679850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-beed1b98-6279-4652-8a04-f619d3029a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:25:07.750106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 591/1175] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:25:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:25:08.018192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742e34ae-e93a-444d-8e6d-4a714c258036 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:08.104288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:08.121100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742e34ae-e93a-444d-8e6d-4a714c258036 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:08.153487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc0cc8d-8e37-472b-8c37-a6196cc3e1a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:08.154106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc0cc8d-8e37-472b-8c37-a6196cc3e1a6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:08.154386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc0cc8d-8e37-472b-8c37-a6196cc3e1a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:08.154657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc0cc8d-8e37-472b-8c37-a6196cc3e1a6 None admin] PATH_INFO: `/v3/domains/d9709e4cb018446cb32b0e57f5198b7b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:08.155752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc0cc8d-8e37-472b-8c37-a6196cc3e1a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=d9709e4cb018446cb32b0e57f5198b7b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:08.157216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc0cc8d-8e37-472b-8c37-a6196cc3e1a6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:08.158755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 585/1176] 10.222.0.25 () {56 vars in 1183 bytes} [Mon Sep 26 01:25:08 2022] GET /identity/v3/domains/d9709e4cb018446cb32b0e57f5198b7b => generated 267 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:08.164598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beed1b98-6279-4652-8a04-f619d3029a9a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:08.182579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:08.261077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beed1b98-6279-4652-8a04-f619d3029a9a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:08.294037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571b5869-8195-463b-a4ec-a018c8c3effd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:08.294563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571b5869-8195-463b-a4ec-a018c8c3effd None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:08.294838 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571b5869-8195-463b-a4ec-a018c8c3effd None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:08.295105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571b5869-8195-463b-a4ec-a018c8c3effd None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:08.295749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571b5869-8195-463b-a4ec-a018c8c3effd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:08.297220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571b5869-8195-463b-a4ec-a018c8c3effd None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:08.319944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 592/1177] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:25:08 2022] POST /identity/v3/users => generated 311 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:25:10.502372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d66a01-1eaa-4fa1-89de-958465c23137 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:10.502953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d66a01-1eaa-4fa1-89de-958465c23137 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:10.502953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d66a01-1eaa-4fa1-89de-958465c23137 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:10.504248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 586/1178] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:25:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:25:10.510328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ae7688-2e55-4301-9f38-aadc620c1869 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:10.510734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ae7688-2e55-4301-9f38-aadc620c1869 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:10.510871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ae7688-2e55-4301-9f38-aadc620c1869 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:10.529182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-97ae7688-2e55-4301-9f38-aadc620c1869 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:25:10.530374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-97ae7688-2e55-4301-9f38-aadc620c1869 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:25:10.594443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 593/1179] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:25:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:25:10.863301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d66a01-1eaa-4fa1-89de-958465c23137 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:10.964412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d66a01-1eaa-4fa1-89de-958465c23137 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:10.996706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadc6ac8-eacc-4c79-87e5-090727a4adf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:10.997310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadc6ac8-eacc-4c79-87e5-090727a4adf4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:10.997614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadc6ac8-eacc-4c79-87e5-090727a4adf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:10.997881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadc6ac8-eacc-4c79-87e5-090727a4adf4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:11.008211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eadc6ac8-eacc-4c79-87e5-090727a4adf4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-eadc6ac8-eacc-4c79-87e5-090727a4adf4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:25:11.012920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:25:11.014105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:25:11.015032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 587/1180] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:25:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:25:11.019905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ae7688-2e55-4301-9f38-aadc620c1869 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:11.023730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:11.117964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ae7688-2e55-4301-9f38-aadc620c1869 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:11.150865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa4ff4d-2165-40a2-9620-053c7934dc26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:11.151463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4ff4d-2165-40a2-9620-053c7934dc26 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:11.151764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4ff4d-2165-40a2-9620-053c7934dc26 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:11.152035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4ff4d-2165-40a2-9620-053c7934dc26 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:11.152721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa4ff4d-2165-40a2-9620-053c7934dc26 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:25:11.153022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa4ff4d-2165-40a2-9620-053c7934dc26 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:11.154374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa4ff4d-2165-40a2-9620-053c7934dc26 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:11.162784 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 594/1181] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:25:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:11.168659 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadc6ac8-eacc-4c79-87e5-090727a4adf4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:11.183652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:11.266946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadc6ac8-eacc-4c79-87e5-090727a4adf4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:11.299875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637598b7-a992-4cf1-a2e0-3cfc5d926184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:11.300518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637598b7-a992-4cf1-a2e0-3cfc5d926184 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:11.300796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637598b7-a992-4cf1-a2e0-3cfc5d926184 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:11.301065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637598b7-a992-4cf1-a2e0-3cfc5d926184 None admin] PATH_INFO: `/v3/users/d7e317b720cd4230ba921ea107b5301e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:11.313639 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637598b7-a992-4cf1-a2e0-3cfc5d926184 None admin] RBAC: Authorizing `identity:get_user(user_id=d7e317b720cd4230ba921ea107b5301e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:11.315079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637598b7-a992-4cf1-a2e0-3cfc5d926184 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:11.316676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 588/1182] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:25:11 2022] GET /identity/v3/users/d7e317b720cd4230ba921ea107b5301e => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:11.323198 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa4ff4d-2165-40a2-9620-053c7934dc26 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:11.386668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:11.420409 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa4ff4d-2165-40a2-9620-053c7934dc26 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:11.452149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b242b91-f4e9-42df-807e-6e0e5c136c3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:11.452757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b242b91-f4e9-42df-807e-6e0e5c136c3e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:11.453033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b242b91-f4e9-42df-807e-6e0e5c136c3e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:11.453304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b242b91-f4e9-42df-807e-6e0e5c136c3e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:11.516994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b242b91-f4e9-42df-807e-6e0e5c136c3e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:11.518413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b242b91-f4e9-42df-807e-6e0e5c136c3e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:11.579313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:11.613782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 595/1183] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:25:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:25:11.620183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637598b7-a992-4cf1-a2e0-3cfc5d926184 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:11.687153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:11.718119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637598b7-a992-4cf1-a2e0-3cfc5d926184 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:11.751064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd65432-38f7-40ae-8a3c-bb52231b8d0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:11.751678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd65432-38f7-40ae-8a3c-bb52231b8d0c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:11.751959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd65432-38f7-40ae-8a3c-bb52231b8d0c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:11.752233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd65432-38f7-40ae-8a3c-bb52231b8d0c None admin] PATH_INFO: `/v3/projects/99b72bfdcacd4b39a11085092c2a4d91` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:11.753365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd65432-38f7-40ae-8a3c-bb52231b8d0c None admin] RBAC: Authorizing `identity:get_project(project_id=99b72bfdcacd4b39a11085092c2a4d91)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:11.754797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd65432-38f7-40ae-8a3c-bb52231b8d0c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:11.756424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 589/1184] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:25:11 2022] GET /identity/v3/projects/99b72bfdcacd4b39a11085092c2a4d91 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:11.762399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b242b91-f4e9-42df-807e-6e0e5c136c3e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:11.856670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b242b91-f4e9-42df-807e-6e0e5c136c3e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:11.889328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bb196b-453a-44f5-bcac-4137ed7936a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:11.889902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bb196b-453a-44f5-bcac-4137ed7936a1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:11.890175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bb196b-453a-44f5-bcac-4137ed7936a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:11.890439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bb196b-453a-44f5-bcac-4137ed7936a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:11.891143 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bb196b-453a-44f5-bcac-4137ed7936a1 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=99b72bfdcacd4b39a11085092c2a4d91, user.id=d7e317b720cd4230ba921ea107b5301e) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:25:11.891447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bb196b-453a-44f5-bcac-4137ed7936a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:11.892877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bb196b-453a-44f5-bcac-4137ed7936a1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:11.898537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 596/1185] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:25:11 2022] GET /identity/v3/role_assignments?user.id=d7e317b720cd4230ba921ea107b5301e&scope.project.id=99b72bfdcacd4b39a11085092c2a4d91&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:14.014164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b177fd1-a084-459a-95ca-fa43d271ed85 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:14.014721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b177fd1-a084-459a-95ca-fa43d271ed85 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:14.014721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b177fd1-a084-459a-95ca-fa43d271ed85 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:14.016029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 590/1186] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:25:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:25:14.021778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70000b15-e8bf-4617-a13c-2c25d93c99ac None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:14.022375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70000b15-e8bf-4617-a13c-2c25d93c99ac None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:14.022375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70000b15-e8bf-4617-a13c-2c25d93c99ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:14.040827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-70000b15-e8bf-4617-a13c-2c25d93c99ac None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:25:14.042015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-70000b15-e8bf-4617-a13c-2c25d93c99ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:25:14.065151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:14.107793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 597/1187] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:25:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:25:14.371836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b177fd1-a084-459a-95ca-fa43d271ed85 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:14.470149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b177fd1-a084-459a-95ca-fa43d271ed85 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:14.503416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a636d30f-9d1a-4652-9feb-3e392776cbd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:14.504028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a636d30f-9d1a-4652-9feb-3e392776cbd1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:14.504308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a636d30f-9d1a-4652-9feb-3e392776cbd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:14.504608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a636d30f-9d1a-4652-9feb-3e392776cbd1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:14.511218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:14.515170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a636d30f-9d1a-4652-9feb-3e392776cbd1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a636d30f-9d1a-4652-9feb-3e392776cbd1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:25:14.520112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:25:14.521321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:25:14.522412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 591/1188] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:25:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:25:14.527404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70000b15-e8bf-4617-a13c-2c25d93c99ac None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:14.622768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70000b15-e8bf-4617-a13c-2c25d93c99ac None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:14.656168 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1997f540-9615-403e-b7c7-3ca8cc58dec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:14.656789 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1997f540-9615-403e-b7c7-3ca8cc58dec5 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:14.657066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1997f540-9615-403e-b7c7-3ca8cc58dec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:14.657336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1997f540-9615-403e-b7c7-3ca8cc58dec5 None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:14.657986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1997f540-9615-403e-b7c7-3ca8cc58dec5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:25:14.658285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1997f540-9615-403e-b7c7-3ca8cc58dec5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:14.659635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1997f540-9615-403e-b7c7-3ca8cc58dec5 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:14.668878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 598/1189] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:25:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:14.676258 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a636d30f-9d1a-4652-9feb-3e392776cbd1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:14.775715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a636d30f-9d1a-4652-9feb-3e392776cbd1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:14.811769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8863e3fa-d0a4-4d20-a8e3-9c688fb0b83b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:14.812407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8863e3fa-d0a4-4d20-a8e3-9c688fb0b83b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:14.812687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8863e3fa-d0a4-4d20-a8e3-9c688fb0b83b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:14.812957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8863e3fa-d0a4-4d20-a8e3-9c688fb0b83b None admin] PATH_INFO: `/v3/users/d7e317b720cd4230ba921ea107b5301e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:14.814070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8863e3fa-d0a4-4d20-a8e3-9c688fb0b83b None admin] RBAC: Authorizing `identity:get_user(user_id=d7e317b720cd4230ba921ea107b5301e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:14.815491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8863e3fa-d0a4-4d20-a8e3-9c688fb0b83b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:14.817076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 592/1190] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:25:14 2022] GET /identity/v3/users/d7e317b720cd4230ba921ea107b5301e => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:14.824049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1997f540-9615-403e-b7c7-3ca8cc58dec5 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:14.857841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:14.939595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1997f540-9615-403e-b7c7-3ca8cc58dec5 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:14.976238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f4e53c-568b-4188-b2b1-aa8672c47b2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:14.976887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f4e53c-568b-4188-b2b1-aa8672c47b2b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:14.977229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f4e53c-568b-4188-b2b1-aa8672c47b2b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:14.977500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f4e53c-568b-4188-b2b1-aa8672c47b2b None admin] PATH_INFO: `/v3/projects/99b72bfdcacd4b39a11085092c2a4d91` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:14.978604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f4e53c-568b-4188-b2b1-aa8672c47b2b None admin] RBAC: Authorizing `identity:get_project(project_id=99b72bfdcacd4b39a11085092c2a4d91)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:14.980023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f4e53c-568b-4188-b2b1-aa8672c47b2b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:14.981661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 599/1191] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:25:14 2022] GET /identity/v3/projects/99b72bfdcacd4b39a11085092c2a4d91 => generated 361 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:14.987696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8863e3fa-d0a4-4d20-a8e3-9c688fb0b83b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:15.065898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:15.107748 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8863e3fa-d0a4-4d20-a8e3-9c688fb0b83b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:15.152213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b41d77-fa6e-42ee-b727-3603054c2577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:15.152895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b41d77-fa6e-42ee-b727-3603054c2577 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:15.153177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b41d77-fa6e-42ee-b727-3603054c2577 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:15.153445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b41d77-fa6e-42ee-b727-3603054c2577 None admin] PATH_INFO: `/v3/projects/99b72bfdcacd4b39a11085092c2a4d91/users/d7e317b720cd4230ba921ea107b5301e/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:15.155346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b41d77-fa6e-42ee-b727-3603054c2577 None admin] RBAC: Authorizing `identity:create_grant(project_id=99b72bfdcacd4b39a11085092c2a4d91, user_id=d7e317b720cd4230ba921ea107b5301e, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:15.156820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b41d77-fa6e-42ee-b727-3603054c2577 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:15.170306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 593/1192] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:25:14 2022] PUT /identity/v3/projects/99b72bfdcacd4b39a11085092c2a4d91/users/d7e317b720cd4230ba921ea107b5301e/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:25:17.343097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65386df-6600-471c-98a8-b71e7dde5b24 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:17.343615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65386df-6600-471c-98a8-b71e7dde5b24 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:17.343615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65386df-6600-471c-98a8-b71e7dde5b24 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:17.344899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 600/1193] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:25:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:25:17.350520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4c9f84-fc73-4a96-b15d-d2550d900487 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:17.350909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4c9f84-fc73-4a96-b15d-d2550d900487 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:17.351063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4c9f84-fc73-4a96-b15d-d2550d900487 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:17.369442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bd4c9f84-fc73-4a96-b15d-d2550d900487 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:25:17.370620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bd4c9f84-fc73-4a96-b15d-d2550d900487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:25:17.434562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 594/1194] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:25:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:25:17.723839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65386df-6600-471c-98a8-b71e7dde5b24 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:17.813878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:17.824294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65386df-6600-471c-98a8-b71e7dde5b24 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:17.857640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abea6195-29d9-4769-a4e1-d7e08fec77d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:17.858253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abea6195-29d9-4769-a4e1-d7e08fec77d0 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:17.858533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abea6195-29d9-4769-a4e1-d7e08fec77d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:17.858804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abea6195-29d9-4769-a4e1-d7e08fec77d0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:17.868680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abea6195-29d9-4769-a4e1-d7e08fec77d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-abea6195-29d9-4769-a4e1-d7e08fec77d0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:25:17.873404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:25:17.874478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:25:17.875359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 601/1195] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:25:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:25:17.880541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4c9f84-fc73-4a96-b15d-d2550d900487 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:17.934269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:17.978283 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4c9f84-fc73-4a96-b15d-d2550d900487 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:18.011899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1021ddd-e084-4fce-9843-111479dba20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:18.012516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1021ddd-e084-4fce-9843-111479dba20b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:18.012792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1021ddd-e084-4fce-9843-111479dba20b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:18.013063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1021ddd-e084-4fce-9843-111479dba20b None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:18.013712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1021ddd-e084-4fce-9843-111479dba20b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:25:18.014016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1021ddd-e084-4fce-9843-111479dba20b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:18.015683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1021ddd-e084-4fce-9843-111479dba20b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:18.023942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 595/1196] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:25:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:18.029853 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abea6195-29d9-4769-a4e1-d7e08fec77d0 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:18.128398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abea6195-29d9-4769-a4e1-d7e08fec77d0 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:18.163152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e078fb0-3c3a-401e-964f-759ea2d16c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:18.163764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e078fb0-3c3a-401e-964f-759ea2d16c92 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:18.164040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e078fb0-3c3a-401e-964f-759ea2d16c92 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:18.164308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e078fb0-3c3a-401e-964f-759ea2d16c92 None admin] PATH_INFO: `/v3/users/d7e317b720cd4230ba921ea107b5301e` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:18.165427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e078fb0-3c3a-401e-964f-759ea2d16c92 None admin] RBAC: Authorizing `identity:get_user(user_id=d7e317b720cd4230ba921ea107b5301e)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:18.166840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e078fb0-3c3a-401e-964f-759ea2d16c92 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:18.168385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 602/1197] 10.222.0.25 () {56 vars in 1177 bytes} [Mon Sep 26 01:25:18 2022] GET /identity/v3/users/d7e317b720cd4230ba921ea107b5301e => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:18.174098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1021ddd-e084-4fce-9843-111479dba20b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:18.272110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1021ddd-e084-4fce-9843-111479dba20b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:18.288669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:18.305517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c71c9a-3236-4ff5-9aad-baf78084a424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:18.306102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c71c9a-3236-4ff5-9aad-baf78084a424 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:18.306379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c71c9a-3236-4ff5-9aad-baf78084a424 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:18.306649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c71c9a-3236-4ff5-9aad-baf78084a424 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:18.372501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c71c9a-3236-4ff5-9aad-baf78084a424 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:18.373931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c71c9a-3236-4ff5-9aad-baf78084a424 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:18.470173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 596/1198] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:25:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:25:18.477509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e078fb0-3c3a-401e-964f-759ea2d16c92 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:18.495946 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:18.573880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e078fb0-3c3a-401e-964f-759ea2d16c92 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:18.607099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df91125-e5c9-4e59-901d-1611e7b147fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:18.607719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df91125-e5c9-4e59-901d-1611e7b147fc None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:18.607990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df91125-e5c9-4e59-901d-1611e7b147fc None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:18.608256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df91125-e5c9-4e59-901d-1611e7b147fc None admin] PATH_INFO: `/v3/projects/99b72bfdcacd4b39a11085092c2a4d91` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:18.609367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df91125-e5c9-4e59-901d-1611e7b147fc None admin] RBAC: Authorizing `identity:get_project(project_id=99b72bfdcacd4b39a11085092c2a4d91)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:18.610776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df91125-e5c9-4e59-901d-1611e7b147fc None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:18.612352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 603/1199] 10.222.0.25 () {56 vars in 1186 bytes} [Mon Sep 26 01:25:18 2022] GET /identity/v3/projects/99b72bfdcacd4b39a11085092c2a4d91 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:18.618481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c71c9a-3236-4ff5-9aad-baf78084a424 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:25:18.636286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:18.716238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c71c9a-3236-4ff5-9aad-baf78084a424 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:25:18.749799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc55418a-f980-4b6f-a427-ef4978c730f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:25:18.750386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc55418a-f980-4b6f-a427-ef4978c730f9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:25:18.750667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc55418a-f980-4b6f-a427-ef4978c730f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:25:18.750942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc55418a-f980-4b6f-a427-ef4978c730f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:25:18.751657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc55418a-f980-4b6f-a427-ef4978c730f9 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=99b72bfdcacd4b39a11085092c2a4d91, user.id=d7e317b720cd4230ba921ea107b5301e) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:25:18.751967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc55418a-f980-4b6f-a427-ef4978c730f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:25:18.753417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc55418a-f980-4b6f-a427-ef4978c730f9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:25:18.759539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 597/1200] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:25:18 2022] GET /identity/v3/role_assignments?user.id=d7e317b720cd4230ba921ea107b5301e&scope.project.id=99b72bfdcacd4b39a11085092c2a4d91&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:25:28.603146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=57 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:25:28.756995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=59 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:14.784302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a33fa63-0474-4a85-8094-608df23e0a8d None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:14.784916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a33fa63-0474-4a85-8094-608df23e0a8d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:14.785338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a33fa63-0474-4a85-8094-608df23e0a8d None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:14.787229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 604/1201] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:26:14.794064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d88c70-1a69-47d9-adab-c37c14660d56 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:14.794552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d88c70-1a69-47d9-adab-c37c14660d56 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:14.794901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d88c70-1a69-47d9-adab-c37c14660d56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:14.816546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-65d88c70-1a69-47d9-adab-c37c14660d56 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:14.817826 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-65d88c70-1a69-47d9-adab-c37c14660d56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:14.908498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 598/1202] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:15.211078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a33fa63-0474-4a85-8094-608df23e0a8d None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:15.359738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a33fa63-0474-4a85-8094-608df23e0a8d None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:15.399382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cc9704-5d0d-4a4a-8ca0-ffb9912cbb3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:15.400127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cc9704-5d0d-4a4a-8ca0-ffb9912cbb3a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:15.400457 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cc9704-5d0d-4a4a-8ca0-ffb9912cbb3a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:15.400689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cc9704-5d0d-4a4a-8ca0-ffb9912cbb3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:15.412739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cc9704-5d0d-4a4a-8ca0-ffb9912cbb3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:15.414282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cc9704-5d0d-4a4a-8ca0-ffb9912cbb3a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-12cc9704-5d0d-4a4a-8ca0-ffb9912cbb3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:15.430923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:15.432085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:15.433444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:15.433444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:15.433444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:15.433444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 605/1203] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:15.438487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d88c70-1a69-47d9-adab-c37c14660d56 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:15.556710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d88c70-1a69-47d9-adab-c37c14660d56 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:15.602309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:15.609725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334bd030-b619-4209-88d9-25e78d991ee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:15.610586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334bd030-b619-4209-88d9-25e78d991ee6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:15.610716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334bd030-b619-4209-88d9-25e78d991ee6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:15.610947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334bd030-b619-4209-88d9-25e78d991ee6 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:15.612235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334bd030-b619-4209-88d9-25e78d991ee6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:15.612667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334bd030-b619-4209-88d9-25e78d991ee6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:15.614286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334bd030-b619-4209-88d9-25e78d991ee6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:15.634707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cc9704-5d0d-4a4a-8ca0-ffb9912cbb3a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:15.636393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 599/1204] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:15.709037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:15.795030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cc9704-5d0d-4a4a-8ca0-ffb9912cbb3a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:15.839994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ad9b3d-a574-403a-b4a4-5438c8f4727b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:15.840737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad9b3d-a574-403a-b4a4-5438c8f4727b None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:15.840975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad9b3d-a574-403a-b4a4-5438c8f4727b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:15.841328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad9b3d-a574-403a-b4a4-5438c8f4727b None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:15.846619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ad9b3d-a574-403a-b4a4-5438c8f4727b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:15.846619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ad9b3d-a574-403a-b4a4-5438c8f4727b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:15.871951 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 606/1205] 10.222.0.25 () {60 vars in 1132 bytes} [Mon Sep 26 01:26:15 2022] POST /identity/v3/users => generated 251 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:26:18.391843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c2bf33-192d-4a5d-851c-81f89a4ccc0d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:18.392397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c2bf33-192d-4a5d-851c-81f89a4ccc0d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:18.392397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c2bf33-192d-4a5d-851c-81f89a4ccc0d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:18.394420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 600/1206] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:26:18.400858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c63cac-42bd-4e56-a310-599b40e5d846 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:18.401287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c63cac-42bd-4e56-a310-599b40e5d846 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:18.401453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c63cac-42bd-4e56-a310-599b40e5d846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:18.423928 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-39c63cac-42bd-4e56-a310-599b40e5d846 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:18.425272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-39c63cac-42bd-4e56-a310-599b40e5d846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:18.496726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 607/1207] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:18.803039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c2bf33-192d-4a5d-851c-81f89a4ccc0d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:18.923495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c2bf33-192d-4a5d-851c-81f89a4ccc0d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:18.964606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148a1771-cd4a-4394-9e91-02021d22c762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:18.965249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148a1771-cd4a-4394-9e91-02021d22c762 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:18.965524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148a1771-cd4a-4394-9e91-02021d22c762 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:18.965807 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148a1771-cd4a-4394-9e91-02021d22c762 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:18.978525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148a1771-cd4a-4394-9e91-02021d22c762 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-148a1771-cd4a-4394-9e91-02021d22c762 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:26:18.984844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:18.985932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:18.986946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 601/1208] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:26:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:18.994568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c63cac-42bd-4e56-a310-599b40e5d846 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:19.038403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:19.107705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c63cac-42bd-4e56-a310-599b40e5d846 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:19.144757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75527c85-71c5-402f-b8b7-dfbcc806080e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:19.145358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75527c85-71c5-402f-b8b7-dfbcc806080e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:19.145638 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75527c85-71c5-402f-b8b7-dfbcc806080e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:19.145911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75527c85-71c5-402f-b8b7-dfbcc806080e None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:19.146569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75527c85-71c5-402f-b8b7-dfbcc806080e None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:19.146904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75527c85-71c5-402f-b8b7-dfbcc806080e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:19.148609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75527c85-71c5-402f-b8b7-dfbcc806080e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:19.157367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 608/1209] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:26:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:19.166314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148a1771-cd4a-4394-9e91-02021d22c762 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:19.229295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:19.303637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148a1771-cd4a-4394-9e91-02021d22c762 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:19.342472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a0f758-4c9b-4c3f-a889-2dd85d7c038a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:19.343075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a0f758-4c9b-4c3f-a889-2dd85d7c038a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:19.343426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a0f758-4c9b-4c3f-a889-2dd85d7c038a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:19.343613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a0f758-4c9b-4c3f-a889-2dd85d7c038a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:19.355188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a0f758-4c9b-4c3f-a889-2dd85d7c038a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:19.356640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a0f758-4c9b-4c3f-a889-2dd85d7c038a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-54a0f758-4c9b-4c3f-a889-2dd85d7c038a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:19.374188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:19.375272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:19.376245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:19.376245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:19.376245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:19.376245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 602/1210] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:26:19.381728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75527c85-71c5-402f-b8b7-dfbcc806080e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:19.486255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:19.490394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75527c85-71c5-402f-b8b7-dfbcc806080e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:19.527834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ae1847-8a1f-4ab1-9c2c-61fc1b136e96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:19.528473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ae1847-8a1f-4ab1-9c2c-61fc1b136e96 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:19.528806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ae1847-8a1f-4ab1-9c2c-61fc1b136e96 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:19.529083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ae1847-8a1f-4ab1-9c2c-61fc1b136e96 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:19.529725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ae1847-8a1f-4ab1-9c2c-61fc1b136e96 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:19.530032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ae1847-8a1f-4ab1-9c2c-61fc1b136e96 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:19.531417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ae1847-8a1f-4ab1-9c2c-61fc1b136e96 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:19.547926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 609/1211] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:19.549307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a0f758-4c9b-4c3f-a889-2dd85d7c038a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:19.688943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a0f758-4c9b-4c3f-a889-2dd85d7c038a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:19.691466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:19.737555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9be399e-882f-4a9b-885c-ced4339a81cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:19.738502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9be399e-882f-4a9b-885c-ced4339a81cc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:19.738773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9be399e-882f-4a9b-885c-ced4339a81cc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:19.739176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9be399e-882f-4a9b-885c-ced4339a81cc None admin] PATH_INFO: `/v3/users/placement` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:19.748081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9be399e-882f-4a9b-885c-ced4339a81cc None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:19.749744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9be399e-882f-4a9b-885c-ced4339a81cc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f9be399e-882f-4a9b-885c-ced4339a81cc None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:26:19.763065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:19.764127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:19.765183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:19.765183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:19.765183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:26:19.765183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:26:19.765183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:26:19.765183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:26:19.765183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:26:19.765183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:26:19.765183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:19.765183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:19.769948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 603/1212] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:19 2022] GET /identity/v3/users/placement => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:19.770932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ae1847-8a1f-4ab1-9c2c-61fc1b136e96 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:19.875221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ae1847-8a1f-4ab1-9c2c-61fc1b136e96 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:19.910936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485c9c17-becc-4863-afd2-8e3ccb8ab929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:19.911508 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485c9c17-becc-4863-afd2-8e3ccb8ab929 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:19.911784 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485c9c17-becc-4863-afd2-8e3ccb8ab929 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:19.912097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485c9c17-becc-4863-afd2-8e3ccb8ab929 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:19.912839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485c9c17-becc-4863-afd2-8e3ccb8ab929 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:19.913140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485c9c17-becc-4863-afd2-8e3ccb8ab929 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:19.914593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485c9c17-becc-4863-afd2-8e3ccb8ab929 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:19.933190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 610/1213] 10.222.0.25 () {56 vars in 1143 bytes} [Mon Sep 26 01:26:19 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:19.939160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9be399e-882f-4a9b-885c-ced4339a81cc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:20.059476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9be399e-882f-4a9b-885c-ced4339a81cc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:20.098097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1056348d-19dd-4777-91e0-e794fcad914f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:20.098854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1056348d-19dd-4777-91e0-e794fcad914f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:20.099135 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1056348d-19dd-4777-91e0-e794fcad914f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:20.099410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1056348d-19dd-4777-91e0-e794fcad914f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:20.133479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:20.177685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1056348d-19dd-4777-91e0-e794fcad914f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:20.179301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1056348d-19dd-4777-91e0-e794fcad914f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:20.293140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 604/1214] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:26:19 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:26:20.298826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485c9c17-becc-4863-afd2-8e3ccb8ab929 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:20.321040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:20.415915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485c9c17-becc-4863-afd2-8e3ccb8ab929 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:20.460266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d2d7b7-3d9b-406c-b420-54a26723c91d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:20.460952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d2d7b7-3d9b-406c-b420-54a26723c91d None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:20.461266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d2d7b7-3d9b-406c-b420-54a26723c91d None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:20.461539 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d2d7b7-3d9b-406c-b420-54a26723c91d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:20.474266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d2d7b7-3d9b-406c-b420-54a26723c91d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:20.475922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d2d7b7-3d9b-406c-b420-54a26723c91d None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-21d2d7b7-3d9b-406c-b420-54a26723c91d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:20.494350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:20.495858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:20.497044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:20.497044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:20.497044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:20.497044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 611/1215] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:20.505256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1056348d-19dd-4777-91e0-e794fcad914f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:20.593004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:20.643437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1056348d-19dd-4777-91e0-e794fcad914f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:20.679063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8e5652-2716-4759-b52d-513799b20088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:20.679665 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8e5652-2716-4759-b52d-513799b20088 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:20.679993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8e5652-2716-4759-b52d-513799b20088 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:20.680310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8e5652-2716-4759-b52d-513799b20088 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:20.680882 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8e5652-2716-4759-b52d-513799b20088 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:20.681185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8e5652-2716-4759-b52d-513799b20088 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:20.682555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8e5652-2716-4759-b52d-513799b20088 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:20.697110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 605/1216] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:20.701707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d2d7b7-3d9b-406c-b420-54a26723c91d None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:20.782609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:20.816510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d2d7b7-3d9b-406c-b420-54a26723c91d None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:20.852079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6629fc7-394e-42a1-b715-ca174853b47e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:20.852660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6629fc7-394e-42a1-b715-ca174853b47e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:20.852939 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6629fc7-394e-42a1-b715-ca174853b47e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:20.853209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6629fc7-394e-42a1-b715-ca174853b47e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:20.859977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6629fc7-394e-42a1-b715-ca174853b47e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:20.862068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6629fc7-394e-42a1-b715-ca174853b47e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-f6629fc7-394e-42a1-b715-ca174853b47e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:26:20.875141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:20.876247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:20.878062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:26:20.878062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:26:20.878062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:26:20.878062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:26:20.878062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:26:20.878062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:26:20.878062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:20.878062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:20.878062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 612/1217] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:26:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:20.883321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8e5652-2716-4759-b52d-513799b20088 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:20.994707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8e5652-2716-4759-b52d-513799b20088 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:21.023546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:21.036078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269b1b5-680c-4a66-a784-27f32e4909b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:21.036686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269b1b5-680c-4a66-a784-27f32e4909b8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:21.037029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269b1b5-680c-4a66-a784-27f32e4909b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:21.037368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269b1b5-680c-4a66-a784-27f32e4909b8 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:21.038060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269b1b5-680c-4a66-a784-27f32e4909b8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:21.038409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269b1b5-680c-4a66-a784-27f32e4909b8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:21.039877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269b1b5-680c-4a66-a784-27f32e4909b8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:21.055431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 606/1218] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:26:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:21.061754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6629fc7-394e-42a1-b715-ca174853b47e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:21.172627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6629fc7-394e-42a1-b715-ca174853b47e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:21.204580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:21.212552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da4b1ff-f9c4-4cb5-ac17-3d7af0995878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:21.213187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da4b1ff-f9c4-4cb5-ac17-3d7af0995878 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:21.213484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da4b1ff-f9c4-4cb5-ac17-3d7af0995878 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:21.213766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da4b1ff-f9c4-4cb5-ac17-3d7af0995878 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:21.214497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da4b1ff-f9c4-4cb5-ac17-3d7af0995878 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=427016f8da514dc4bbed21de08207dbb) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:21.214808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da4b1ff-f9c4-4cb5-ac17-3d7af0995878 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:21.216398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da4b1ff-f9c4-4cb5-ac17-3d7af0995878 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:21.222899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 613/1219] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:26:21 2022] GET /identity/v3/role_assignments?user.id=427016f8da514dc4bbed21de08207dbb&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:26:23.373322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e5c2f2-9153-458c-86c1-11ae608022a7 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:23.373883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e5c2f2-9153-458c-86c1-11ae608022a7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:23.373883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e5c2f2-9153-458c-86c1-11ae608022a7 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:23.375128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 607/1220] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:26:23.380572 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cf17a0-14ed-43d5-8c79-f921a1bea12e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:23.381010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cf17a0-14ed-43d5-8c79-f921a1bea12e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:23.381121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cf17a0-14ed-43d5-8c79-f921a1bea12e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:23.400336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-74cf17a0-14ed-43d5-8c79-f921a1bea12e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:23.401584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-74cf17a0-14ed-43d5-8c79-f921a1bea12e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:23.487979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 614/1221] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:23.760985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e5c2f2-9153-458c-86c1-11ae608022a7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:23.880438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e5c2f2-9153-458c-86c1-11ae608022a7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:23.920637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39d39a8-bd1e-45c3-8004-8758c826c6ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:23.921269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39d39a8-bd1e-45c3-8004-8758c826c6ab None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:23.921548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39d39a8-bd1e-45c3-8004-8758c826c6ab None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:23.921818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39d39a8-bd1e-45c3-8004-8758c826c6ab None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:23.932950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39d39a8-bd1e-45c3-8004-8758c826c6ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-c39d39a8-bd1e-45c3-8004-8758c826c6ab None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:26:23.937846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:23.939149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:23.940623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 608/1222] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:26:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:23.945520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cf17a0-14ed-43d5-8c79-f921a1bea12e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:24.060326 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cf17a0-14ed-43d5-8c79-f921a1bea12e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:24.085215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:24.098959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188bc401-5f20-452c-a8bf-fb45185e326b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:24.099699 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188bc401-5f20-452c-a8bf-fb45185e326b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:24.100002 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188bc401-5f20-452c-a8bf-fb45185e326b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:24.100309 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188bc401-5f20-452c-a8bf-fb45185e326b None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:24.101104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188bc401-5f20-452c-a8bf-fb45185e326b None admin] RBAC: Adding query filter params (name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:24.101546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188bc401-5f20-452c-a8bf-fb45185e326b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:24.103134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188bc401-5f20-452c-a8bf-fb45185e326b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:24.113399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 615/1223] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:26:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:24.119235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39d39a8-bd1e-45c3-8004-8758c826c6ab None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:24.165486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:24.230656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39d39a8-bd1e-45c3-8004-8758c826c6ab None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:24.272575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f20e2cf-cac0-4b75-9ec2-0d864df60da7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:24.273342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f20e2cf-cac0-4b75-9ec2-0d864df60da7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:24.273660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f20e2cf-cac0-4b75-9ec2-0d864df60da7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:24.273929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f20e2cf-cac0-4b75-9ec2-0d864df60da7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:24.285206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f20e2cf-cac0-4b75-9ec2-0d864df60da7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:24.286800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f20e2cf-cac0-4b75-9ec2-0d864df60da7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4f20e2cf-cac0-4b75-9ec2-0d864df60da7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:24.304952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:24.306397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:24.307552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:24.307552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:24.307552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:24.307552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 609/1224] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:24.312088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188bc401-5f20-452c-a8bf-fb45185e326b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:24.429792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188bc401-5f20-452c-a8bf-fb45185e326b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:24.469388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbda6cf-7b18-4770-809f-bdd7128c6281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:24.470060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbda6cf-7b18-4770-809f-bdd7128c6281 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:24.470349 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbda6cf-7b18-4770-809f-bdd7128c6281 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:24.471207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbda6cf-7b18-4770-809f-bdd7128c6281 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:24.471954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbda6cf-7b18-4770-809f-bdd7128c6281 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:24.472257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbda6cf-7b18-4770-809f-bdd7128c6281 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:24.473796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbda6cf-7b18-4770-809f-bdd7128c6281 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:24.487067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 616/1225] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:24.493444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f20e2cf-cac0-4b75-9ec2-0d864df60da7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:24.598534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:24.607320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f20e2cf-cac0-4b75-9ec2-0d864df60da7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:24.648277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859a21d1-fcc9-4b6e-8565-36a89e3c5b02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:24.649036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a21d1-fcc9-4b6e-8565-36a89e3c5b02 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:24.649329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a21d1-fcc9-4b6e-8565-36a89e3c5b02 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:24.649606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a21d1-fcc9-4b6e-8565-36a89e3c5b02 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:24.658819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859a21d1-fcc9-4b6e-8565-36a89e3c5b02 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:24.660496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859a21d1-fcc9-4b6e-8565-36a89e3c5b02 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-859a21d1-fcc9-4b6e-8565-36a89e3c5b02 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:26:24.672461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:24.673524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:24.674988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 610/1226] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:24 2022] GET /identity/v3/users/placement => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:24.679589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbda6cf-7b18-4770-809f-bdd7128c6281 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:24.735707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:24.793596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbda6cf-7b18-4770-809f-bdd7128c6281 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:24.832813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef59f1de-ffd7-46df-ba11-16e2a0627728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:24.833577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef59f1de-ffd7-46df-ba11-16e2a0627728 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:24.833894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef59f1de-ffd7-46df-ba11-16e2a0627728 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:24.834197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef59f1de-ffd7-46df-ba11-16e2a0627728 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:24.835137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef59f1de-ffd7-46df-ba11-16e2a0627728 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:24.835484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef59f1de-ffd7-46df-ba11-16e2a0627728 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:24.837201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef59f1de-ffd7-46df-ba11-16e2a0627728 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:24.861662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 617/1227] 10.222.0.25 () {56 vars in 1143 bytes} [Mon Sep 26 01:26:24 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:24.862812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859a21d1-fcc9-4b6e-8565-36a89e3c5b02 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:24.971666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859a21d1-fcc9-4b6e-8565-36a89e3c5b02 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:25.008576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee299c80-8bfb-4ce3-b3f6-b02f8e4ab80b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:25.009541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee299c80-8bfb-4ce3-b3f6-b02f8e4ab80b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:25.009541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee299c80-8bfb-4ce3-b3f6-b02f8e4ab80b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:25.009768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee299c80-8bfb-4ce3-b3f6-b02f8e4ab80b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:25.022706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee299c80-8bfb-4ce3-b3f6-b02f8e4ab80b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:25.024181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee299c80-8bfb-4ce3-b3f6-b02f8e4ab80b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ee299c80-8bfb-4ce3-b3f6-b02f8e4ab80b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:25.039303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:25.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:25.045895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:25.045895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:25.045895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:25.045895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 611/1228] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:26:25.048043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef59f1de-ffd7-46df-ba11-16e2a0627728 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:25.153315 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef59f1de-ffd7-46df-ba11-16e2a0627728 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:25.155560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:25.188615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41c2a75-9079-4e11-b7ca-13d1e6ef5aed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:25.189230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c2a75-9079-4e11-b7ca-13d1e6ef5aed None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:25.190083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c2a75-9079-4e11-b7ca-13d1e6ef5aed None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:25.190485 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c2a75-9079-4e11-b7ca-13d1e6ef5aed None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:25.191137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41c2a75-9079-4e11-b7ca-13d1e6ef5aed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:25.191442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41c2a75-9079-4e11-b7ca-13d1e6ef5aed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:25.192877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41c2a75-9079-4e11-b7ca-13d1e6ef5aed None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:25.204373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 618/1229] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:26:25.211033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee299c80-8bfb-4ce3-b3f6-b02f8e4ab80b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:25.220790 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:25.317403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee299c80-8bfb-4ce3-b3f6-b02f8e4ab80b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:25.352557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d7a5f5-e58c-4408-8737-c96c195a93da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:25.353195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d7a5f5-e58c-4408-8737-c96c195a93da None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:25.353473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d7a5f5-e58c-4408-8737-c96c195a93da None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:25.353741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d7a5f5-e58c-4408-8737-c96c195a93da None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:25.360881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d7a5f5-e58c-4408-8737-c96c195a93da None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:25.363078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d7a5f5-e58c-4408-8737-c96c195a93da None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-49d7a5f5-e58c-4408-8737-c96c195a93da None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:26:25.375259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:25.376232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:25.378063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:26:25.378063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:26:25.378063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:26:25.378063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:26:25.378063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:26:25.378063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:26:25.378063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:25.378063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:25.378063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 612/1230] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:26:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:25.382849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41c2a75-9079-4e11-b7ca-13d1e6ef5aed None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:25.498263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41c2a75-9079-4e11-b7ca-13d1e6ef5aed None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:25.534701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1013365-0433-441d-ac94-8a22742b908c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:25.535320 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1013365-0433-441d-ac94-8a22742b908c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:25.535601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1013365-0433-441d-ac94-8a22742b908c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:25.535880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1013365-0433-441d-ac94-8a22742b908c None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:25.536613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1013365-0433-441d-ac94-8a22742b908c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:25.536940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1013365-0433-441d-ac94-8a22742b908c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:25.538350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1013365-0433-441d-ac94-8a22742b908c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:25.552148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 619/1231] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:26:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:25.558443 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d7a5f5-e58c-4408-8737-c96c195a93da None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:25.633119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:25.672197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d7a5f5-e58c-4408-8737-c96c195a93da None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:25.710849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d595dc-adf3-435c-a5a0-68cd71747616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:25.711517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d595dc-adf3-435c-a5a0-68cd71747616 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:25.711800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d595dc-adf3-435c-a5a0-68cd71747616 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:25.712128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d595dc-adf3-435c-a5a0-68cd71747616 None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/427016f8da514dc4bbed21de08207dbb/roles/03dc4e232ede47a699064dda38338428` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:25.728480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d595dc-adf3-435c-a5a0-68cd71747616 None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=427016f8da514dc4bbed21de08207dbb, role_id=03dc4e232ede47a699064dda38338428)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:25.729962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d595dc-adf3-435c-a5a0-68cd71747616 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:25.748472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 613/1232] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:26:25 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/427016f8da514dc4bbed21de08207dbb/roles/03dc4e232ede47a699064dda38338428 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:26:27.946867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5cea76-82fa-47a2-87c8-bda3e80fdf69 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:27.947435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5cea76-82fa-47a2-87c8-bda3e80fdf69 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:27.947435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5cea76-82fa-47a2-87c8-bda3e80fdf69 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:27.948998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 620/1233] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:26:27.954241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf17226-d5d4-4f51-8d26-48c0f0fc86ad None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:27.954668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf17226-d5d4-4f51-8d26-48c0f0fc86ad None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:27.954876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf17226-d5d4-4f51-8d26-48c0f0fc86ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:27.976533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bbf17226-d5d4-4f51-8d26-48c0f0fc86ad None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:27.977956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bbf17226-d5d4-4f51-8d26-48c0f0fc86ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:28.052124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 614/1234] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:28.368022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5cea76-82fa-47a2-87c8-bda3e80fdf69 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:28.395291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:28.481011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5cea76-82fa-47a2-87c8-bda3e80fdf69 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:28.518053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35975967-eb87-4582-b3f3-4ef1acb5e263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:28.518683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35975967-eb87-4582-b3f3-4ef1acb5e263 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:28.518960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35975967-eb87-4582-b3f3-4ef1acb5e263 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:28.519231 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35975967-eb87-4582-b3f3-4ef1acb5e263 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:28.529527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35975967-eb87-4582-b3f3-4ef1acb5e263 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-35975967-eb87-4582-b3f3-4ef1acb5e263 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:26:28.534384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:28.536660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:28.537690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 621/1235] 10.222.0.25 () {56 vars in 1102 bytes} [Mon Sep 26 01:26:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:28.542478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf17226-d5d4-4f51-8d26-48c0f0fc86ad None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:28.592610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:28.651929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf17226-d5d4-4f51-8d26-48c0f0fc86ad None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:28.690607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b44e88-e0e4-4d6b-beee-64530b7a13f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:28.691231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b44e88-e0e4-4d6b-beee-64530b7a13f0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:28.691471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b44e88-e0e4-4d6b-beee-64530b7a13f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:28.691692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b44e88-e0e4-4d6b-beee-64530b7a13f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:28.692528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b44e88-e0e4-4d6b-beee-64530b7a13f0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:28.692720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b44e88-e0e4-4d6b-beee-64530b7a13f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:28.694106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b44e88-e0e4-4d6b-beee-64530b7a13f0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:28.702914 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 615/1236] 10.222.0.25 () {56 vars in 1103 bytes} [Mon Sep 26 01:26:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:28.711407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35975967-eb87-4582-b3f3-4ef1acb5e263 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:28.810309 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:28.829705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35975967-eb87-4582-b3f3-4ef1acb5e263 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:28.869178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ea81ee-3dcb-454b-a4ab-ae2547950994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:28.869826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ea81ee-3dcb-454b-a4ab-ae2547950994 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:28.870108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ea81ee-3dcb-454b-a4ab-ae2547950994 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:28.870383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ea81ee-3dcb-454b-a4ab-ae2547950994 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:28.881269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ea81ee-3dcb-454b-a4ab-ae2547950994 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:28.882732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ea81ee-3dcb-454b-a4ab-ae2547950994 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-e0ea81ee-3dcb-454b-a4ab-ae2547950994 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:28.897520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:28.898760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:28.899677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:28.899677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:28.899677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:28.899677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 622/1237] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:28.904778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b44e88-e0e4-4d6b-beee-64530b7a13f0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:29.033562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b44e88-e0e4-4d6b-beee-64530b7a13f0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:29.048190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:29.077772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e1b6bd-e957-4edb-9a41-6e7a30a8aa4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:29.078442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e1b6bd-e957-4edb-9a41-6e7a30a8aa4f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:29.078726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e1b6bd-e957-4edb-9a41-6e7a30a8aa4f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:29.079036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e1b6bd-e957-4edb-9a41-6e7a30a8aa4f None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:29.079743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e1b6bd-e957-4edb-9a41-6e7a30a8aa4f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:29.080055 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e1b6bd-e957-4edb-9a41-6e7a30a8aa4f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:29.081861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e1b6bd-e957-4edb-9a41-6e7a30a8aa4f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:29.094559 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 616/1238] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:29.100403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ea81ee-3dcb-454b-a4ab-ae2547950994 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:29.212978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ea81ee-3dcb-454b-a4ab-ae2547950994 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:29.248856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3066c1da-c757-45d9-a170-0d854ad139b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:29.249501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3066c1da-c757-45d9-a170-0d854ad139b1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:29.249793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3066c1da-c757-45d9-a170-0d854ad139b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:29.250060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3066c1da-c757-45d9-a170-0d854ad139b1 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:29.253134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:29.258416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3066c1da-c757-45d9-a170-0d854ad139b1 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:29.260128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3066c1da-c757-45d9-a170-0d854ad139b1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-3066c1da-c757-45d9-a170-0d854ad139b1 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:26:29.271530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:29.275311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:29.276543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 623/1239] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:29 2022] GET /identity/v3/users/placement => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:29.278522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e1b6bd-e957-4edb-9a41-6e7a30a8aa4f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:29.389212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e1b6bd-e957-4edb-9a41-6e7a30a8aa4f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:29.425564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231406f5-22f8-4d71-a983-7ceb8cd9fced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:29.426167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231406f5-22f8-4d71-a983-7ceb8cd9fced None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:29.426450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231406f5-22f8-4d71-a983-7ceb8cd9fced None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:29.426718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231406f5-22f8-4d71-a983-7ceb8cd9fced None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:29.427446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231406f5-22f8-4d71-a983-7ceb8cd9fced None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:29.427750 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231406f5-22f8-4d71-a983-7ceb8cd9fced None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:29.429233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231406f5-22f8-4d71-a983-7ceb8cd9fced None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:29.447084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 617/1240] 10.222.0.25 () {56 vars in 1143 bytes} [Mon Sep 26 01:26:29 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:29.453081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3066c1da-c757-45d9-a170-0d854ad139b1 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:29.556822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3066c1da-c757-45d9-a170-0d854ad139b1 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:29.592529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c33169-8c9d-4593-84d3-c3ec515fb845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:29.593138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c33169-8c9d-4593-84d3-c3ec515fb845 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:29.593419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c33169-8c9d-4593-84d3-c3ec515fb845 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:29.593689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c33169-8c9d-4593-84d3-c3ec515fb845 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:29.660570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:29.667378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c33169-8c9d-4593-84d3-c3ec515fb845 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:29.668864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c33169-8c9d-4593-84d3-c3ec515fb845 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:29.789879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 624/1241] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:26:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 26 01:26:29.791352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231406f5-22f8-4d71-a983-7ceb8cd9fced None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:29.824828 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:29.906879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231406f5-22f8-4d71-a983-7ceb8cd9fced None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:29.949300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c7478e-a686-4da8-901a-932ca1e3ad1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:29.949930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c7478e-a686-4da8-901a-932ca1e3ad1d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:29.950204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c7478e-a686-4da8-901a-932ca1e3ad1d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:29.950479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c7478e-a686-4da8-901a-932ca1e3ad1d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:29.961946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c7478e-a686-4da8-901a-932ca1e3ad1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:29.963621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c7478e-a686-4da8-901a-932ca1e3ad1d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-37c7478e-a686-4da8-901a-932ca1e3ad1d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:29.979956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:29.981202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:29.982365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:29.982365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:29.982365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:29.982365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 618/1242] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:29.988023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c33169-8c9d-4593-84d3-c3ec515fb845 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:30.095772 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:30.102149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c33169-8c9d-4593-84d3-c3ec515fb845 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:30.140172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e68c64-fdbf-405f-95f7-0deb6aeaf843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:30.140814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e68c64-fdbf-405f-95f7-0deb6aeaf843 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:30.141092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e68c64-fdbf-405f-95f7-0deb6aeaf843 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:30.141364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e68c64-fdbf-405f-95f7-0deb6aeaf843 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:30.142001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e68c64-fdbf-405f-95f7-0deb6aeaf843 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:30.142338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e68c64-fdbf-405f-95f7-0deb6aeaf843 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:30.143765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e68c64-fdbf-405f-95f7-0deb6aeaf843 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:30.155368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 625/1243] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:30.161784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c7478e-a686-4da8-901a-932ca1e3ad1d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:30.246424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:30.270997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c7478e-a686-4da8-901a-932ca1e3ad1d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:30.308317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafacabb-7983-4385-b214-bfdf2bfc115d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:30.309059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafacabb-7983-4385-b214-bfdf2bfc115d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:30.309410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafacabb-7983-4385-b214-bfdf2bfc115d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:30.309757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafacabb-7983-4385-b214-bfdf2bfc115d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:30.317588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafacabb-7983-4385-b214-bfdf2bfc115d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:30.319048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafacabb-7983-4385-b214-bfdf2bfc115d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-cafacabb-7983-4385-b214-bfdf2bfc115d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:26:30.331579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:30.332799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:30.334123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:26:30.334123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:26:30.334123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:26:30.334123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:26:30.334123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:26:30.334123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:26:30.334123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:30.334123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:30.334123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 619/1244] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:26:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:30.338742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e68c64-fdbf-405f-95f7-0deb6aeaf843 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:30.453319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e68c64-fdbf-405f-95f7-0deb6aeaf843 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:30.489468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09ac5fc-22cf-440a-9f2f-f11814ee32ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:30.490102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ac5fc-22cf-440a-9f2f-f11814ee32ea None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:30.490378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ac5fc-22cf-440a-9f2f-f11814ee32ea None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:30.490652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ac5fc-22cf-440a-9f2f-f11814ee32ea None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:30.491354 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09ac5fc-22cf-440a-9f2f-f11814ee32ea None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:30.491656 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09ac5fc-22cf-440a-9f2f-f11814ee32ea None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:30.493143 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09ac5fc-22cf-440a-9f2f-f11814ee32ea None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:30.506400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 626/1245] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:26:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:30.512625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafacabb-7983-4385-b214-bfdf2bfc115d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:30.623301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafacabb-7983-4385-b214-bfdf2bfc115d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:30.658191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75745be4-5197-4409-81c5-d4217ba7a7f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:30.658965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75745be4-5197-4409-81c5-d4217ba7a7f3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:30.659245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75745be4-5197-4409-81c5-d4217ba7a7f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:30.659537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75745be4-5197-4409-81c5-d4217ba7a7f3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:30.660241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75745be4-5197-4409-81c5-d4217ba7a7f3 None admin] RBAC: Adding query filter params (role.id=03dc4e232ede47a699064dda38338428, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=427016f8da514dc4bbed21de08207dbb) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:30.660577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75745be4-5197-4409-81c5-d4217ba7a7f3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:30.662015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75745be4-5197-4409-81c5-d4217ba7a7f3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:30.668505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 620/1246] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:26:30 2022] GET /identity/v3/role_assignments?user.id=427016f8da514dc4bbed21de08207dbb&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=03dc4e232ede47a699064dda38338428 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:32.933154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39497094-9a87-4f72-a417-9c4e36210227 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:32.933700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39497094-9a87-4f72-a417-9c4e36210227 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:32.933700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39497094-9a87-4f72-a417-9c4e36210227 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:32.935187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 627/1247] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:26:32.941272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0db144c-0a04-4cf2-926d-ea87b0442035 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:32.941797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0db144c-0a04-4cf2-926d-ea87b0442035 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:32.942036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0db144c-0a04-4cf2-926d-ea87b0442035 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:32.949241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:32.963205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b0db144c-0a04-4cf2-926d-ea87b0442035 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:32.964497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b0db144c-0a04-4cf2-926d-ea87b0442035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:33.044168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 621/1248] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:33.319824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39497094-9a87-4f72-a417-9c4e36210227 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:33.339445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:33.434714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39497094-9a87-4f72-a417-9c4e36210227 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:33.474926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5457b2a-c09c-4c3d-b44b-e22db92c0e82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:33.475548 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5457b2a-c09c-4c3d-b44b-e22db92c0e82 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:33.475827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5457b2a-c09c-4c3d-b44b-e22db92c0e82 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:33.476098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5457b2a-c09c-4c3d-b44b-e22db92c0e82 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:33.486629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5457b2a-c09c-4c3d-b44b-e22db92c0e82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-f5457b2a-c09c-4c3d-b44b-e22db92c0e82 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:26:33.492060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:33.493275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:33.494526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 628/1249] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:26:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:33.499708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0db144c-0a04-4cf2-926d-ea87b0442035 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:33.611806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0db144c-0a04-4cf2-926d-ea87b0442035 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:33.645133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:33.650311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba93c92-0622-4cde-90ae-d433dee28ca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:33.651116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba93c92-0622-4cde-90ae-d433dee28ca6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:33.651415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba93c92-0622-4cde-90ae-d433dee28ca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:33.651824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba93c92-0622-4cde-90ae-d433dee28ca6 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:33.652544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba93c92-0622-4cde-90ae-d433dee28ca6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:33.652848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba93c92-0622-4cde-90ae-d433dee28ca6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:33.654216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba93c92-0622-4cde-90ae-d433dee28ca6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:33.663167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 622/1250] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:26:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:33.669178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5457b2a-c09c-4c3d-b44b-e22db92c0e82 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:33.761722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:33.787647 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5457b2a-c09c-4c3d-b44b-e22db92c0e82 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:33.826554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae7d452-b721-4e31-b1bb-1ff8c4928f14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:33.827185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae7d452-b721-4e31-b1bb-1ff8c4928f14 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:33.827469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae7d452-b721-4e31-b1bb-1ff8c4928f14 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:33.827747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae7d452-b721-4e31-b1bb-1ff8c4928f14 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:33.838953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae7d452-b721-4e31-b1bb-1ff8c4928f14 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:33.840562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae7d452-b721-4e31-b1bb-1ff8c4928f14 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-6ae7d452-b721-4e31-b1bb-1ff8c4928f14 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:33.857567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:33.858782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:33.859913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:33.859913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:33.859913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:33.860517 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 629/1251] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 26 01:26:33.865493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba93c92-0622-4cde-90ae-d433dee28ca6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:33.973707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba93c92-0622-4cde-90ae-d433dee28ca6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:34.009555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f14e67-12b6-43bf-97f9-17f8a5f27268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:34.010169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f14e67-12b6-43bf-97f9-17f8a5f27268 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:34.010444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f14e67-12b6-43bf-97f9-17f8a5f27268 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:34.010714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f14e67-12b6-43bf-97f9-17f8a5f27268 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:34.011367 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f14e67-12b6-43bf-97f9-17f8a5f27268 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:34.011668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f14e67-12b6-43bf-97f9-17f8a5f27268 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:34.013091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f14e67-12b6-43bf-97f9-17f8a5f27268 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:34.026715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 623/1252] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:34.032687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae7d452-b721-4e31-b1bb-1ff8c4928f14 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:34.138915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae7d452-b721-4e31-b1bb-1ff8c4928f14 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:34.167125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:34.176243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c20009a-50db-49be-a105-f56d57b763f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:34.176924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c20009a-50db-49be-a105-f56d57b763f2 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:34.177205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c20009a-50db-49be-a105-f56d57b763f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:34.177479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c20009a-50db-49be-a105-f56d57b763f2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:34.185473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c20009a-50db-49be-a105-f56d57b763f2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:34.186970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c20009a-50db-49be-a105-f56d57b763f2 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-8c20009a-50db-49be-a105-f56d57b763f2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:26:34.199063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:34.200162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:34.201175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 630/1253] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:34 2022] GET /identity/v3/users/placement => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:34.207085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f14e67-12b6-43bf-97f9-17f8a5f27268 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:34.264853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:34.310242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f14e67-12b6-43bf-97f9-17f8a5f27268 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:34.345883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b64277-f531-4365-a690-b4c87084560a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:34.346486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b64277-f531-4365-a690-b4c87084560a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:34.346761 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b64277-f531-4365-a690-b4c87084560a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:34.347027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b64277-f531-4365-a690-b4c87084560a None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:34.347755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b64277-f531-4365-a690-b4c87084560a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:34.348056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b64277-f531-4365-a690-b4c87084560a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:34.349536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b64277-f531-4365-a690-b4c87084560a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:34.368138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 624/1254] 10.222.0.25 () {56 vars in 1143 bytes} [Mon Sep 26 01:26:34 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:34.373720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c20009a-50db-49be-a105-f56d57b763f2 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:34.476940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c20009a-50db-49be-a105-f56d57b763f2 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:34.511302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ed3f36-b740-44e3-beb7-4e2969b47a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:34.511902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ed3f36-b740-44e3-beb7-4e2969b47a81 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:34.512177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ed3f36-b740-44e3-beb7-4e2969b47a81 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:34.512475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ed3f36-b740-44e3-beb7-4e2969b47a81 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:34.566867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:34.579262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ed3f36-b740-44e3-beb7-4e2969b47a81 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:34.582112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ed3f36-b740-44e3-beb7-4e2969b47a81 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:34.678339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 631/1255] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:26:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:26:34.684293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b64277-f531-4365-a690-b4c87084560a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:34.793876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b64277-f531-4365-a690-b4c87084560a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:34.807130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:34.828078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470b53d-f43f-448d-a522-88e1b26fde47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:34.828718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470b53d-f43f-448d-a522-88e1b26fde47 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:34.828993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470b53d-f43f-448d-a522-88e1b26fde47 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:34.829261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470b53d-f43f-448d-a522-88e1b26fde47 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:34.840091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0470b53d-f43f-448d-a522-88e1b26fde47 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:34.841558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0470b53d-f43f-448d-a522-88e1b26fde47 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0470b53d-f43f-448d-a522-88e1b26fde47 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:34.855907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:34.857067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:34.858268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:34.858268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:34.858268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:34.858268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 625/1256] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:34.863301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ed3f36-b740-44e3-beb7-4e2969b47a81 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:34.947606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:34.960442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ed3f36-b740-44e3-beb7-4e2969b47a81 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:34.993520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d961170b-0ed0-461e-90e2-f45efacfa32b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:34.994110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d961170b-0ed0-461e-90e2-f45efacfa32b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:34.994385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d961170b-0ed0-461e-90e2-f45efacfa32b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:34.994657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d961170b-0ed0-461e-90e2-f45efacfa32b None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:34.995293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d961170b-0ed0-461e-90e2-f45efacfa32b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:34.995594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d961170b-0ed0-461e-90e2-f45efacfa32b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:34.996975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d961170b-0ed0-461e-90e2-f45efacfa32b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:35.007833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 632/1257] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:35.013530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470b53d-f43f-448d-a522-88e1b26fde47 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:35.113798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470b53d-f43f-448d-a522-88e1b26fde47 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:35.146842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d259e1-b9da-44be-bf19-60a474abbb12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:35.147460 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d259e1-b9da-44be-bf19-60a474abbb12 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:35.147742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d259e1-b9da-44be-bf19-60a474abbb12 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:35.148021 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d259e1-b9da-44be-bf19-60a474abbb12 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:35.154503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d259e1-b9da-44be-bf19-60a474abbb12 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:35.155941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d259e1-b9da-44be-bf19-60a474abbb12 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-91d259e1-b9da-44be-bf19-60a474abbb12 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:26:35.167799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:35.168783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:35.169689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:26:35.169689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:26:35.169689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:26:35.169689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:26:35.169689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:26:35.169689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:26:35.169689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:35.169689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:35.169689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 626/1258] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:26:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:35.175043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d961170b-0ed0-461e-90e2-f45efacfa32b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:35.271839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d961170b-0ed0-461e-90e2-f45efacfa32b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:35.303515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f9ebd7-8c7d-4970-940b-cab81794f10f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:35.304043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f9ebd7-8c7d-4970-940b-cab81794f10f None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:35.304322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f9ebd7-8c7d-4970-940b-cab81794f10f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:35.304623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f9ebd7-8c7d-4970-940b-cab81794f10f None admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:35.305298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f9ebd7-8c7d-4970-940b-cab81794f10f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:35.305599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f9ebd7-8c7d-4970-940b-cab81794f10f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:35.307016 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f9ebd7-8c7d-4970-940b-cab81794f10f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:35.318958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 633/1259] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:26:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:35.324943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d259e1-b9da-44be-bf19-60a474abbb12 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:35.350814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:35.422902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d259e1-b9da-44be-bf19-60a474abbb12 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:35.456210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3292450-d752-4bae-b8a0-d72d161660c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:35.456819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3292450-d752-4bae-b8a0-d72d161660c0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:35.457100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3292450-d752-4bae-b8a0-d72d161660c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:35.457373 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3292450-d752-4bae-b8a0-d72d161660c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:35.458085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3292450-d752-4bae-b8a0-d72d161660c0 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=427016f8da514dc4bbed21de08207dbb) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:35.458395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3292450-d752-4bae-b8a0-d72d161660c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:35.459820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3292450-d752-4bae-b8a0-d72d161660c0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:35.465140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 627/1260] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:26:35 2022] GET /identity/v3/role_assignments?user.id=427016f8da514dc4bbed21de08207dbb&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:37.577912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e996f3b5-a9c8-4690-a855-6e82b8fee4a6 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:37.578440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e996f3b5-a9c8-4690-a855-6e82b8fee4a6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:37.578440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e996f3b5-a9c8-4690-a855-6e82b8fee4a6 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:37.581066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 634/1261] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:26:37.586303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c1733a-3523-415e-8e71-225caa7049dc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:37.586739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c1733a-3523-415e-8e71-225caa7049dc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:37.586859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c1733a-3523-415e-8e71-225caa7049dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:37.604762 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e8c1733a-3523-415e-8e71-225caa7049dc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:37.605950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e8c1733a-3523-415e-8e71-225caa7049dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:37.670469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 628/1262] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:37.958842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e996f3b5-a9c8-4690-a855-6e82b8fee4a6 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:37.971330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:38.059069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e996f3b5-a9c8-4690-a855-6e82b8fee4a6 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:38.092098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574ff250-bba3-468d-8094-3b41456bbe9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:38.092741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574ff250-bba3-468d-8094-3b41456bbe9a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:38.093021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574ff250-bba3-468d-8094-3b41456bbe9a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:38.093290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574ff250-bba3-468d-8094-3b41456bbe9a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:38.103562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574ff250-bba3-468d-8094-3b41456bbe9a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-574ff250-bba3-468d-8094-3b41456bbe9a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:26:38.108286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:38.109399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:38.110448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 635/1263] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:26:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:38.115545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c1733a-3523-415e-8e71-225caa7049dc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:38.133378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:38.212651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c1733a-3523-415e-8e71-225caa7049dc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:38.246378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866a837e-ee6b-442d-9a02-8b48610319cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:38.246971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866a837e-ee6b-442d-9a02-8b48610319cc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:38.247251 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866a837e-ee6b-442d-9a02-8b48610319cc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:38.247526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866a837e-ee6b-442d-9a02-8b48610319cc None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:38.248183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866a837e-ee6b-442d-9a02-8b48610319cc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:38.248518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866a837e-ee6b-442d-9a02-8b48610319cc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:38.249896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866a837e-ee6b-442d-9a02-8b48610319cc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:38.258250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 629/1264] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:26:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:38.263954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574ff250-bba3-468d-8094-3b41456bbe9a None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:38.330742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:38.362006 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574ff250-bba3-468d-8094-3b41456bbe9a None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:38.395698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bdd479-a531-4ff5-825c-67bc2ae6040e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:38.396304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bdd479-a531-4ff5-825c-67bc2ae6040e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:38.396613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bdd479-a531-4ff5-825c-67bc2ae6040e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:38.396886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bdd479-a531-4ff5-825c-67bc2ae6040e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:38.406466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bdd479-a531-4ff5-825c-67bc2ae6040e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:38.407923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bdd479-a531-4ff5-825c-67bc2ae6040e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-f2bdd479-a531-4ff5-825c-67bc2ae6040e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:38.422012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:38.423114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:38.424473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:38.424473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:38.424473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:38.424473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 636/1265] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:38.428957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866a837e-ee6b-442d-9a02-8b48610319cc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:38.508179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:38.527555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866a837e-ee6b-442d-9a02-8b48610319cc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:38.559735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ed87d2-0780-4931-ac20-947f2de761f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:38.560324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ed87d2-0780-4931-ac20-947f2de761f3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:38.560643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ed87d2-0780-4931-ac20-947f2de761f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:38.560923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ed87d2-0780-4931-ac20-947f2de761f3 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:38.561558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ed87d2-0780-4931-ac20-947f2de761f3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:38.561885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ed87d2-0780-4931-ac20-947f2de761f3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:38.563283 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ed87d2-0780-4931-ac20-947f2de761f3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:38.575315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 630/1266] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:38.581985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bdd479-a531-4ff5-825c-67bc2ae6040e None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:38.679031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bdd479-a531-4ff5-825c-67bc2ae6040e None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:38.700203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:38.716680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b97f3-44c9-43c2-9b47-8e1ea218d84b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:38.717305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b97f3-44c9-43c2-9b47-8e1ea218d84b None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:38.717619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b97f3-44c9-43c2-9b47-8e1ea218d84b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:38.717863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b97f3-44c9-43c2-9b47-8e1ea218d84b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:38.726274 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38b97f3-44c9-43c2-9b47-8e1ea218d84b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:38.727747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38b97f3-44c9-43c2-9b47-8e1ea218d84b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-b38b97f3-44c9-43c2-9b47-8e1ea218d84b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:26:38.739376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:38.740557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:38.741794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 637/1267] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:38 2022] GET /identity/v3/users/placement => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:38.746353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ed87d2-0780-4931-ac20-947f2de761f3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:38.843505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ed87d2-0780-4931-ac20-947f2de761f3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:38.876866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90245cab-ff2a-400f-b01d-738b49c374e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:38.877450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90245cab-ff2a-400f-b01d-738b49c374e3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:38.877729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90245cab-ff2a-400f-b01d-738b49c374e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:38.878000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90245cab-ff2a-400f-b01d-738b49c374e3 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:38.878719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90245cab-ff2a-400f-b01d-738b49c374e3 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:38.879024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90245cab-ff2a-400f-b01d-738b49c374e3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:38.880463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90245cab-ff2a-400f-b01d-738b49c374e3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:38.888854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:38.898313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 631/1268] 10.222.0.25 () {56 vars in 1143 bytes} [Mon Sep 26 01:26:38 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:38.904155 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b97f3-44c9-43c2-9b47-8e1ea218d84b None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:39.000517 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b97f3-44c9-43c2-9b47-8e1ea218d84b None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:39.033314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3794ad49-7120-4363-8047-fab12781df80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:39.033916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3794ad49-7120-4363-8047-fab12781df80 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:39.034189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3794ad49-7120-4363-8047-fab12781df80 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:39.034457 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3794ad49-7120-4363-8047-fab12781df80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:39.044786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3794ad49-7120-4363-8047-fab12781df80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:39.046229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3794ad49-7120-4363-8047-fab12781df80 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-3794ad49-7120-4363-8047-fab12781df80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:39.060161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:39.061331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:39.062456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:39.062456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:39.062456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:39.062456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 638/1269] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:39.067983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90245cab-ff2a-400f-b01d-738b49c374e3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:39.167778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90245cab-ff2a-400f-b01d-738b49c374e3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:39.200691 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efdcfe-1472-4510-94c2-677bc4981971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:39.201285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efdcfe-1472-4510-94c2-677bc4981971 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:39.201565 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efdcfe-1472-4510-94c2-677bc4981971 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:39.201840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efdcfe-1472-4510-94c2-677bc4981971 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:39.202506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71efdcfe-1472-4510-94c2-677bc4981971 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:39.202810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71efdcfe-1472-4510-94c2-677bc4981971 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:39.204778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71efdcfe-1472-4510-94c2-677bc4981971 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:39.216182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 632/1270] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:39.221995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3794ad49-7120-4363-8047-fab12781df80 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:39.253513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:39.320700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3794ad49-7120-4363-8047-fab12781df80 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:39.353777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f303a2c-aa65-417b-9f3c-9a743fcb522c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:39.354389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f303a2c-aa65-417b-9f3c-9a743fcb522c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:39.354672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f303a2c-aa65-417b-9f3c-9a743fcb522c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:39.354948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f303a2c-aa65-417b-9f3c-9a743fcb522c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:39.361449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f303a2c-aa65-417b-9f3c-9a743fcb522c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:39.362887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f303a2c-aa65-417b-9f3c-9a743fcb522c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-6f303a2c-aa65-417b-9f3c-9a743fcb522c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:26:39.373847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:39.374905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:39.376060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:26:39.376060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:26:39.376060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:26:39.376060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:26:39.376060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:26:39.376060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:26:39.376060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:39.376060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:39.376060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 639/1271] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:26:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:39.380785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efdcfe-1472-4510-94c2-677bc4981971 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:39.437715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:39.479198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efdcfe-1472-4510-94c2-677bc4981971 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:39.511271 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f0102d-a4a5-4407-beea-c43a36d1b181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:39.511861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f0102d-a4a5-4407-beea-c43a36d1b181 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:39.512142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f0102d-a4a5-4407-beea-c43a36d1b181 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:39.512445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f0102d-a4a5-4407-beea-c43a36d1b181 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:39.513129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f0102d-a4a5-4407-beea-c43a36d1b181 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:39.513436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f0102d-a4a5-4407-beea-c43a36d1b181 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:39.514858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f0102d-a4a5-4407-beea-c43a36d1b181 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:39.527043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 633/1272] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:26:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:39.533577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f303a2c-aa65-417b-9f3c-9a743fcb522c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:39.622518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:39.631269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f303a2c-aa65-417b-9f3c-9a743fcb522c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:39.664132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0a280c-f7ce-4c68-936d-73f86f5c1f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:39.664791 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a280c-f7ce-4c68-936d-73f86f5c1f3b None admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:39.665086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a280c-f7ce-4c68-936d-73f86f5c1f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:39.665337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a280c-f7ce-4c68-936d-73f86f5c1f3b None admin] PATH_INFO: `/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/427016f8da514dc4bbed21de08207dbb/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:39.667163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0a280c-f7ce-4c68-936d-73f86f5c1f3b None admin] RBAC: Authorizing `identity:create_grant(project_id=205f567457bc42f9b1a00721cfd6c41f, user_id=427016f8da514dc4bbed21de08207dbb, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:39.668602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0a280c-f7ce-4c68-936d-73f86f5c1f3b None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:39.680620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 640/1273] 10.222.0.25 () {58 vars in 1439 bytes} [Mon Sep 26 01:26:39 2022] PUT /identity/v3/projects/205f567457bc42f9b1a00721cfd6c41f/users/427016f8da514dc4bbed21de08207dbb/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:26:41.847116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bedf758-b13d-4512-aa7f-5124ff6a6180 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:41.847776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bedf758-b13d-4512-aa7f-5124ff6a6180 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:41.847776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bedf758-b13d-4512-aa7f-5124ff6a6180 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:41.849094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 634/1274] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:26:41.854639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4693c625-f40e-49b3-8865-bab9d5c11c20 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:41.855055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4693c625-f40e-49b3-8865-bab9d5c11c20 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:41.855167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4693c625-f40e-49b3-8865-bab9d5c11c20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:41.874382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-4693c625-f40e-49b3-8865-bab9d5c11c20 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:41.875942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4693c625-f40e-49b3-8865-bab9d5c11c20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:41.944156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 641/1275] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:42.212199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bedf758-b13d-4512-aa7f-5124ff6a6180 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:42.312533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bedf758-b13d-4512-aa7f-5124ff6a6180 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:42.329394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:42.346910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66e923f-f122-49b0-9631-7826b03b6a25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:42.347527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66e923f-f122-49b0-9631-7826b03b6a25 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:42.347805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66e923f-f122-49b0-9631-7826b03b6a25 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:42.348073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66e923f-f122-49b0-9631-7826b03b6a25 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:42.357926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66e923f-f122-49b0-9631-7826b03b6a25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a66e923f-f122-49b0-9631-7826b03b6a25 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 26 01:26:42.363261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:42.364439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:42.365362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 635/1276] 10.222.0.25 () {56 vars in 1096 bytes} [Mon Sep 26 01:26:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:42.370341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4693c625-f40e-49b3-8865-bab9d5c11c20 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:42.447327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:42.466069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4693c625-f40e-49b3-8865-bab9d5c11c20 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:42.498940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30290516-cc71-4291-99ed-65333de0689c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:42.499526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30290516-cc71-4291-99ed-65333de0689c None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:42.499802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30290516-cc71-4291-99ed-65333de0689c None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:42.500073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30290516-cc71-4291-99ed-65333de0689c None admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:42.500798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30290516-cc71-4291-99ed-65333de0689c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:42.501107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30290516-cc71-4291-99ed-65333de0689c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:42.502443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30290516-cc71-4291-99ed-65333de0689c None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:42.510145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 642/1277] 10.222.0.25 () {56 vars in 1099 bytes} [Mon Sep 26 01:26:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:42.515914 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66e923f-f122-49b0-9631-7826b03b6a25 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:42.614406 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66e923f-f122-49b0-9631-7826b03b6a25 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:42.647682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce6e646-2f1e-4063-92cf-630148420213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:42.648291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce6e646-2f1e-4063-92cf-630148420213 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:42.648599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce6e646-2f1e-4063-92cf-630148420213 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:42.648871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce6e646-2f1e-4063-92cf-630148420213 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:42.658888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce6e646-2f1e-4063-92cf-630148420213 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:42.660336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce6e646-2f1e-4063-92cf-630148420213 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4ce6e646-2f1e-4063-92cf-630148420213 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:42.676324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:42.677578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:42.678801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:42.678801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:42.678801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:42.678801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 636/1278] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:42.683138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30290516-cc71-4291-99ed-65333de0689c None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:42.781446 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30290516-cc71-4291-99ed-65333de0689c None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:42.814159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faf9c26-d3e8-4933-ac63-3e6a4f4e70e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:42.814897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faf9c26-d3e8-4933-ac63-3e6a4f4e70e1 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:42.815184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faf9c26-d3e8-4933-ac63-3e6a4f4e70e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:42.815461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faf9c26-d3e8-4933-ac63-3e6a4f4e70e1 None admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:42.816099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faf9c26-d3e8-4933-ac63-3e6a4f4e70e1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:42.817426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faf9c26-d3e8-4933-ac63-3e6a4f4e70e1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:42.818787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faf9c26-d3e8-4933-ac63-3e6a4f4e70e1 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:42.830710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 643/1279] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:42.836425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce6e646-2f1e-4063-92cf-630148420213 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:42.867604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:42.935531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce6e646-2f1e-4063-92cf-630148420213 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:42.969372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c40986-fbf6-4bf5-a6d7-b22274356ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:42.970056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c40986-fbf6-4bf5-a6d7-b22274356ecf None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:42.970333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c40986-fbf6-4bf5-a6d7-b22274356ecf None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:42.970604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c40986-fbf6-4bf5-a6d7-b22274356ecf None admin] PATH_INFO: `/v3/users/placement` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:42.977727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c40986-fbf6-4bf5-a6d7-b22274356ecf None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:42.979154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c40986-fbf6-4bf5-a6d7-b22274356ecf None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-24c40986-fbf6-4bf5-a6d7-b22274356ecf None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 26 01:26:42.990039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:42.991167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:42.992551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 637/1280] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:42 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:42.996879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faf9c26-d3e8-4933-ac63-3e6a4f4e70e1 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:43.000579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:43.094922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faf9c26-d3e8-4933-ac63-3e6a4f4e70e1 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:43.128266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55146004-7e08-40a6-a2cd-40311a7d06f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:43.128901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55146004-7e08-40a6-a2cd-40311a7d06f5 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:43.129182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55146004-7e08-40a6-a2cd-40311a7d06f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:43.129457 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55146004-7e08-40a6-a2cd-40311a7d06f5 None admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:43.130177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55146004-7e08-40a6-a2cd-40311a7d06f5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:43.130482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55146004-7e08-40a6-a2cd-40311a7d06f5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:43.131907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55146004-7e08-40a6-a2cd-40311a7d06f5 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:43.150042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 644/1281] 10.222.0.25 () {56 vars in 1143 bytes} [Mon Sep 26 01:26:42 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:43.155777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c40986-fbf6-4bf5-a6d7-b22274356ecf None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:43.245063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:43.255485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c40986-fbf6-4bf5-a6d7-b22274356ecf None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:43.288972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cef2e2-12fb-49de-a455-fa2c36fc45fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:43.289557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cef2e2-12fb-49de-a455-fa2c36fc45fc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:43.289832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cef2e2-12fb-49de-a455-fa2c36fc45fc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:43.290102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cef2e2-12fb-49de-a455-fa2c36fc45fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:43.355062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cef2e2-12fb-49de-a455-fa2c36fc45fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:43.356454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cef2e2-12fb-49de-a455-fa2c36fc45fc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:43.443256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:43.453467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 638/1282] 10.222.0.25 () {58 vars in 1282 bytes} [Mon Sep 26 01:26:43 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 26 01:26:43.460171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55146004-7e08-40a6-a2cd-40311a7d06f5 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:43.520760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:43.557710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55146004-7e08-40a6-a2cd-40311a7d06f5 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:43.592249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173166ea-4136-4afd-98cd-4ce566068fa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:43.592885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173166ea-4136-4afd-98cd-4ce566068fa8 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:43.593162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173166ea-4136-4afd-98cd-4ce566068fa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:43.593429 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173166ea-4136-4afd-98cd-4ce566068fa8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:43.603081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173166ea-4136-4afd-98cd-4ce566068fa8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:43.604554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173166ea-4136-4afd-98cd-4ce566068fa8 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-173166ea-4136-4afd-98cd-4ce566068fa8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 26 01:26:43.618689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:43.619797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 26 01:26:43.620768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 26 01:26:43.620768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 26 01:26:43.620768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:43.620768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 645/1283] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:26:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:43.625489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cef2e2-12fb-49de-a455-fa2c36fc45fc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:43.725102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cef2e2-12fb-49de-a455-fa2c36fc45fc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:43.758205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d28b69f-0d46-4452-8b9a-e8943f85564c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:43.758797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d28b69f-0d46-4452-8b9a-e8943f85564c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:43.759077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d28b69f-0d46-4452-8b9a-e8943f85564c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:43.759347 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d28b69f-0d46-4452-8b9a-e8943f85564c None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:43.759979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d28b69f-0d46-4452-8b9a-e8943f85564c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:43.760280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d28b69f-0d46-4452-8b9a-e8943f85564c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:43.761673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d28b69f-0d46-4452-8b9a-e8943f85564c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:43.772805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 639/1284] 10.222.0.25 () {56 vars in 1109 bytes} [Mon Sep 26 01:26:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:26:43.778589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173166ea-4136-4afd-98cd-4ce566068fa8 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:43.876010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173166ea-4136-4afd-98cd-4ce566068fa8 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:43.889015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:43.908930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72396065-d138-4e75-98e3-c9bfdb2655eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:43.909540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72396065-d138-4e75-98e3-c9bfdb2655eb None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:43.909820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72396065-d138-4e75-98e3-c9bfdb2655eb None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:43.910098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72396065-d138-4e75-98e3-c9bfdb2655eb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:43.916421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72396065-d138-4e75-98e3-c9bfdb2655eb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:43.917845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72396065-d138-4e75-98e3-c9bfdb2655eb None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application [None req-72396065-d138-4e75-98e3-c9bfdb2655eb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 26 01:26:43.929161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:43.930238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:43.931300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 26 01:26:43.931300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 26 01:26:43.931300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 26 01:26:43.931300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 26 01:26:43.931300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 26 01:26:43.931300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 26 01:26:43.931300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 26 01:26:43.931300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: ERROR keystone.server.flask.application Sep 26 01:26:43.931300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 646/1285] 10.222.0.25 () {56 vars in 1111 bytes} [Mon Sep 26 01:26:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:43.936107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d28b69f-0d46-4452-8b9a-e8943f85564c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:43.984915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:44.034072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d28b69f-0d46-4452-8b9a-e8943f85564c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:44.067570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4fc82c-12b7-43f0-9c22-4709c1b8ed1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:44.068163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4fc82c-12b7-43f0-9c22-4709c1b8ed1b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:44.068472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4fc82c-12b7-43f0-9c22-4709c1b8ed1b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:44.068741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4fc82c-12b7-43f0-9c22-4709c1b8ed1b None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:44.069418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4fc82c-12b7-43f0-9c22-4709c1b8ed1b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:44.069718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4fc82c-12b7-43f0-9c22-4709c1b8ed1b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:44.071125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4fc82c-12b7-43f0-9c22-4709c1b8ed1b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:44.083450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 640/1286] 10.222.0.25 () {56 vars in 1148 bytes} [Mon Sep 26 01:26:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:44.089437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72396065-d138-4e75-98e3-c9bfdb2655eb None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:44.186339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72396065-d138-4e75-98e3-c9bfdb2655eb None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:44.218823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f8a0af-1a35-4bd3-8f9d-e534e2fde946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:44.219920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f8a0af-1a35-4bd3-8f9d-e534e2fde946 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:44.220200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f8a0af-1a35-4bd3-8f9d-e534e2fde946 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:44.220507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f8a0af-1a35-4bd3-8f9d-e534e2fde946 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:44.221219 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f8a0af-1a35-4bd3-8f9d-e534e2fde946 None admin] RBAC: Adding query filter params (role.id=3d9e246c2bd145e0a09d90666bad9c59, scope.project.id=205f567457bc42f9b1a00721cfd6c41f, user.id=427016f8da514dc4bbed21de08207dbb) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:44.221525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f8a0af-1a35-4bd3-8f9d-e534e2fde946 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:44.222927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f8a0af-1a35-4bd3-8f9d-e534e2fde946 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:44.228431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 647/1287] 10.222.0.25 () {56 vars in 1374 bytes} [Mon Sep 26 01:26:44 2022] GET /identity/v3/role_assignments?user.id=427016f8da514dc4bbed21de08207dbb&scope.project.id=205f567457bc42f9b1a00721cfd6c41f&role.id=3d9e246c2bd145e0a09d90666bad9c59 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:46.368798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2733dc-55dc-4d9b-8dcf-0584b0378c97 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:46.369356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2733dc-55dc-4d9b-8dcf-0584b0378c97 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:46.369356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2733dc-55dc-4d9b-8dcf-0584b0378c97 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:46.370581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 641/1288] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:26:46.376241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56778ed6-3073-437c-8bc6-da97dd0d108a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:46.376850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56778ed6-3073-437c-8bc6-da97dd0d108a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:46.376850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56778ed6-3073-437c-8bc6-da97dd0d108a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:46.394760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-56778ed6-3073-437c-8bc6-da97dd0d108a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:46.395950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-56778ed6-3073-437c-8bc6-da97dd0d108a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:46.425063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:46.462140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 648/1289] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:46.733466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2733dc-55dc-4d9b-8dcf-0584b0378c97 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:46.843618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2733dc-55dc-4d9b-8dcf-0584b0378c97 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:46.854542 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:46.877979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8def8f28-0d40-45e6-9244-36aff957771d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:46.878594 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8def8f28-0d40-45e6-9244-36aff957771d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:46.878872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8def8f28-0d40-45e6-9244-36aff957771d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:46.879144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8def8f28-0d40-45e6-9244-36aff957771d None admin] PATH_INFO: `/v3/services/placement` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:46.879718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8def8f28-0d40-45e6-9244-36aff957771d None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:46.881103 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8def8f28-0d40-45e6-9244-36aff957771d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-8def8f28-0d40-45e6-9244-36aff957771d None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:46.889783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:46.891020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:46.892829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 642/1290] 10.222.0.25 () {56 vars in 1117 bytes} [Mon Sep 26 01:26:46 2022] GET /identity/v3/services/placement => generated 90 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:46.897326 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56778ed6-3073-437c-8bc6-da97dd0d108a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:46.992126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56778ed6-3073-437c-8bc6-da97dd0d108a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:47.024639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f5cdeb-4fa5-4a6b-9db2-b53c862f0c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:47.025224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f5cdeb-4fa5-4a6b-9db2-b53c862f0c9e None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:47.025502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f5cdeb-4fa5-4a6b-9db2-b53c862f0c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:47.025774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f5cdeb-4fa5-4a6b-9db2-b53c862f0c9e None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:47.026395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f5cdeb-4fa5-4a6b-9db2-b53c862f0c9e None admin] RBAC: Adding query filter params (name=placement) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:47.026696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f5cdeb-4fa5-4a6b-9db2-b53c862f0c9e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:47.028040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f5cdeb-4fa5-4a6b-9db2-b53c862f0c9e None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:47.033331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 649/1291] 10.222.0.25 () {56 vars in 1116 bytes} [Mon Sep 26 01:26:46 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:47.038968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8def8f28-0d40-45e6-9244-36aff957771d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:47.136491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8def8f28-0d40-45e6-9244-36aff957771d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:47.169591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4614c8f-bcad-474c-bd0f-35afb626abef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:47.170172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4614c8f-bcad-474c-bd0f-35afb626abef None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:47.170450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4614c8f-bcad-474c-bd0f-35afb626abef None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:47.170721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4614c8f-bcad-474c-bd0f-35afb626abef None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:47.171336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4614c8f-bcad-474c-bd0f-35afb626abef None admin] RBAC: Adding query filter params (type=placement) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:47.171636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4614c8f-bcad-474c-bd0f-35afb626abef None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:47.173129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4614c8f-bcad-474c-bd0f-35afb626abef None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:47.177348 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 643/1292] 10.222.0.25 () {56 vars in 1116 bytes} [Mon Sep 26 01:26:47 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:49.298403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba1650d-fd61-447a-9a13-a38368148def None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:49.298998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba1650d-fd61-447a-9a13-a38368148def None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:49.298998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba1650d-fd61-447a-9a13-a38368148def None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:49.300267 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 650/1293] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:26:49.305367 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fe4964-fa66-4ac6-953b-4824f9fd3829 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:49.305754 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fe4964-fa66-4ac6-953b-4824f9fd3829 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:49.305916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fe4964-fa66-4ac6-953b-4824f9fd3829 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:49.323719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a4fe4964-fa66-4ac6-953b-4824f9fd3829 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:49.324931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a4fe4964-fa66-4ac6-953b-4824f9fd3829 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:49.354743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:49.392841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 644/1294] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:49.659270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba1650d-fd61-447a-9a13-a38368148def None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:49.689955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:49.759514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba1650d-fd61-447a-9a13-a38368148def None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:49.792657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc1014c-34f7-4061-b241-55747b5e452f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:49.793247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc1014c-34f7-4061-b241-55747b5e452f None admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:49.793528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc1014c-34f7-4061-b241-55747b5e452f None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:49.793798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc1014c-34f7-4061-b241-55747b5e452f None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:49.794448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc1014c-34f7-4061-b241-55747b5e452f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:49.795798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc1014c-34f7-4061-b241-55747b5e452f None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:49.809598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 651/1295] 10.222.0.25 () {60 vars in 1141 bytes} [Mon Sep 26 01:26:49 2022] POST /identity/v3/services => generated 246 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:26:51.929587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207302f1-dda9-4eba-b263-3b73d3162105 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:51.930153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207302f1-dda9-4eba-b263-3b73d3162105 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:51.930153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207302f1-dda9-4eba-b263-3b73d3162105 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:51.931485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 645/1296] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:26:51.936546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9f4e65-8e71-40b2-835a-3904475acc6b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:51.936963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9f4e65-8e71-40b2-835a-3904475acc6b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:51.937091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9f4e65-8e71-40b2-835a-3904475acc6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:51.956364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-af9f4e65-8e71-40b2-835a-3904475acc6b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:51.957569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-af9f4e65-8e71-40b2-835a-3904475acc6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:52.026727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 652/1297] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:51 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:52.296505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207302f1-dda9-4eba-b263-3b73d3162105 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:52.395528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207302f1-dda9-4eba-b263-3b73d3162105 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:52.428280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b64f8f-3636-4e17-a589-9abda38ed07c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:52.428921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b64f8f-3636-4e17-a589-9abda38ed07c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:52.429201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b64f8f-3636-4e17-a589-9abda38ed07c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:52.429475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b64f8f-3636-4e17-a589-9abda38ed07c None admin] PATH_INFO: `/v3/services/placement` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:52.430054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b64f8f-3636-4e17-a589-9abda38ed07c None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:52.431410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b64f8f-3636-4e17-a589-9abda38ed07c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-c5b64f8f-3636-4e17-a589-9abda38ed07c None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:52.440129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:52.441334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:52.442399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 646/1298] 10.222.0.25 () {56 vars in 1117 bytes} [Mon Sep 26 01:26:52 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:52.447195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9f4e65-8e71-40b2-835a-3904475acc6b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:52.465653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:52.545067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9f4e65-8e71-40b2-835a-3904475acc6b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:52.577497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f956c91b-69d9-4a56-8be9-719e137959b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:52.578030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f956c91b-69d9-4a56-8be9-719e137959b6 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:52.578841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f956c91b-69d9-4a56-8be9-719e137959b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:52.579350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f956c91b-69d9-4a56-8be9-719e137959b6 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:52.580262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f956c91b-69d9-4a56-8be9-719e137959b6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:52.580590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f956c91b-69d9-4a56-8be9-719e137959b6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:52.581934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f956c91b-69d9-4a56-8be9-719e137959b6 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:52.586831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 653/1299] 10.222.0.25 () {56 vars in 1116 bytes} [Mon Sep 26 01:26:52 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:52.592709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b64f8f-3636-4e17-a589-9abda38ed07c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:52.623114 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:52.690285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b64f8f-3636-4e17-a589-9abda38ed07c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:52.727302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00112d3f-6f4a-4095-ac97-0bfc875b0d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:52.727902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00112d3f-6f4a-4095-ac97-0bfc875b0d9e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:52.728183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00112d3f-6f4a-4095-ac97-0bfc875b0d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:52.728491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00112d3f-6f4a-4095-ac97-0bfc875b0d9e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:52.729185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00112d3f-6f4a-4095-ac97-0bfc875b0d9e None admin] RBAC: Adding query filter params (interface=public, service_id=0e589736025d4975a42788aeb1bb0c93, region_id=RegionOne) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:52.729491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00112d3f-6f4a-4095-ac97-0bfc875b0d9e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:52.730852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00112d3f-6f4a-4095-ac97-0bfc875b0d9e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:52.736115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 647/1300] 10.222.0.25 () {56 vars in 1251 bytes} [Mon Sep 26 01:26:52 2022] GET /identity/v3/endpoints?service_id=0e589736025d4975a42788aeb1bb0c93&interface=public®ion_id=RegionOne => generated 194 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:52.742402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f956c91b-69d9-4a56-8be9-719e137959b6 None admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:52.826916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:52.846124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f956c91b-69d9-4a56-8be9-719e137959b6 None admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:52.878518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08b245d-81eb-44c8-9de0-16a17e8b8879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:52.879113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08b245d-81eb-44c8-9de0-16a17e8b8879 None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:52.879394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08b245d-81eb-44c8-9de0-16a17e8b8879 None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:52.879669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08b245d-81eb-44c8-9de0-16a17e8b8879 None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:52.880272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08b245d-81eb-44c8-9de0-16a17e8b8879 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:52.881671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08b245d-81eb-44c8-9de0-16a17e8b8879 None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:52.888150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 654/1301] 10.222.0.25 () {56 vars in 1087 bytes} [Mon Sep 26 01:26:52 2022] GET /identity/v3/services => generated 2191 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:26:55.018030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da193b83-dd37-4a8e-b32b-1a4b97ed2caf None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:55.018591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da193b83-dd37-4a8e-b32b-1a4b97ed2caf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:55.018591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da193b83-dd37-4a8e-b32b-1a4b97ed2caf None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:55.019901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 648/1302] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:26:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:26:55.025000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072e6d4-759b-4fab-8f66-ae0332537907 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:55.025401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072e6d4-759b-4fab-8f66-ae0332537907 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:55.025541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072e6d4-759b-4fab-8f66-ae0332537907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:55.044416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-4072e6d4-759b-4fab-8f66-ae0332537907 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:26:55.045615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4072e6d4-759b-4fab-8f66-ae0332537907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:26:55.112439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 655/1303] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:26:55 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 26 01:26:55.383003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da193b83-dd37-4a8e-b32b-1a4b97ed2caf None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:55.476305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:55.483128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da193b83-dd37-4a8e-b32b-1a4b97ed2caf None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:55.516325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678c13be-23f2-40cd-9a46-80b1571d3567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:55.516963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678c13be-23f2-40cd-9a46-80b1571d3567 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:55.517240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678c13be-23f2-40cd-9a46-80b1571d3567 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:55.517515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678c13be-23f2-40cd-9a46-80b1571d3567 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:55.518091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678c13be-23f2-40cd-9a46-80b1571d3567 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:55.519445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678c13be-23f2-40cd-9a46-80b1571d3567 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-678c13be-23f2-40cd-9a46-80b1571d3567 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 26 01:26:55.528054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 26 01:26:55.529250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 26 01:26:55.530349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 649/1304] 10.222.0.25 () {56 vars in 1117 bytes} [Mon Sep 26 01:26:55 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 26 01:26:55.535281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4072e6d4-759b-4fab-8f66-ae0332537907 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:55.609080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:26:55.631479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4072e6d4-759b-4fab-8f66-ae0332537907 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:55.664264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a5190d-3be8-474c-8839-3158be7c465a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:55.664830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5190d-3be8-474c-8839-3158be7c465a None admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:55.665112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5190d-3be8-474c-8839-3158be7c465a None admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:55.665386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5190d-3be8-474c-8839-3158be7c465a None admin] PATH_INFO: `/v3/services` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:55.666019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a5190d-3be8-474c-8839-3158be7c465a None admin] RBAC: Adding query filter params (name=placement) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:26:55.666323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a5190d-3be8-474c-8839-3158be7c465a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:55.667676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a5190d-3be8-474c-8839-3158be7c465a None admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:55.672743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 656/1305] 10.222.0.25 () {56 vars in 1116 bytes} [Mon Sep 26 01:26:55 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:26:55.678447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678c13be-23f2-40cd-9a46-80b1571d3567 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:26:55.780922 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678c13be-23f2-40cd-9a46-80b1571d3567 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:26:55.813794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe043c5e-0a5d-413b-b18e-f3a4e907f31f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:26:55.814384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe043c5e-0a5d-413b-b18e-f3a4e907f31f None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:26:55.814664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe043c5e-0a5d-413b-b18e-f3a4e907f31f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:26:55.814937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe043c5e-0a5d-413b-b18e-f3a4e907f31f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:26:55.815592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe043c5e-0a5d-413b-b18e-f3a4e907f31f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:26:55.816978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe043c5e-0a5d-413b-b18e-f3a4e907f31f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:26:55.836485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 650/1306] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:26:55 2022] POST /identity/v3/endpoints => generated 331 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:27:05.670094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=30 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:27:05.832455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:00.317340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893aac2-40aa-4a25-bf96-59dcb514c106 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:00.317846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893aac2-40aa-4a25-bf96-59dcb514c106 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:00.318086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893aac2-40aa-4a25-bf96-59dcb514c106 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:00.327018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2163daa-a508-4478-9626-0936af1fa566 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:00.327488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2163daa-a508-4478-9626-0936af1fa566 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:00.327783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2163daa-a508-4478-9626-0936af1fa566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:00.330726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 657/1307] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:00 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:00.425591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d2163daa-a508-4478-9626-0936af1fa566 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:00.427133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d2163daa-a508-4478-9626-0936af1fa566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:00.560766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 651/1308] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:00.851276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dac7c9-39a9-4ee8-b062-2b11a6fcdb14 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:00.851646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dac7c9-39a9-4ee8-b062-2b11a6fcdb14 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:00.851832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dac7c9-39a9-4ee8-b062-2b11a6fcdb14 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:00.853571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 658/1309] 10.222.0.25 () {54 vars in 964 bytes} [Mon Sep 26 01:28:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:00.860829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f35192-0f68-4bc5-83fb-38886b382ddf None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:00.861589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f35192-0f68-4bc5-83fb-38886b382ddf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:00.862487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f35192-0f68-4bc5-83fb-38886b382ddf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:00.958110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-63f35192-0f68-4bc5-83fb-38886b382ddf None None] MFA Rules not processed for user `0478ebab37fb4f488eef84f7a241e206`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:00.959470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-63f35192-0f68-4bc5-83fb-38886b382ddf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:00.985366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:01.047733 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 652/1310] 10.222.0.25 () {58 vars in 1061 bytes} [Mon Sep 26 01:28:00 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:01.057380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dac7c9-39a9-4ee8-b062-2b11a6fcdb14 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:01.060188 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-85dac7c9-39a9-4ee8-b062-2b11a6fcdb14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:01.194959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dac7c9-39a9-4ee8-b062-2b11a6fcdb14 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:01.282445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fcb150-0ae5-4b06-af71-d598189e6cee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:01.283210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fcb150-0ae5-4b06-af71-d598189e6cee service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:01.283498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fcb150-0ae5-4b06-af71-d598189e6cee service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:01.283772 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fcb150-0ae5-4b06-af71-d598189e6cee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:01.285430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c9fcb150-0ae5-4b06-af71-d598189e6cee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:01.348079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:01.412994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fcb150-0ae5-4b06-af71-d598189e6cee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:01.414392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fcb150-0ae5-4b06-af71-d598189e6cee service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:01.435123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 659/1311] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:03.973393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae0a87a-ab81-44c4-9b66-1b40b0455ee5 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:03.973929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae0a87a-ab81-44c4-9b66-1b40b0455ee5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:03.973929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae0a87a-ab81-44c4-9b66-1b40b0455ee5 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:03.975272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 653/1312] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:03.980677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0237bf9-8fba-4153-a717-54a881672e19 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:03.981084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0237bf9-8fba-4153-a717-54a881672e19 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:03.981226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0237bf9-8fba-4153-a717-54a881672e19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:04.062512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b0237bf9-8fba-4153-a717-54a881672e19 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:04.063755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b0237bf9-8fba-4153-a717-54a881672e19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:04.160724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 660/1313] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:04.444634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae0a87a-ab81-44c4-9b66-1b40b0455ee5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:04.446177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3ae0a87a-ab81-44c4-9b66-1b40b0455ee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:04.548321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae0a87a-ab81-44c4-9b66-1b40b0455ee5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:04.618713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:04.621151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23066907-5ec2-4bb2-90f5-39b813174213 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:04.621929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23066907-5ec2-4bb2-90f5-39b813174213 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:04.622268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23066907-5ec2-4bb2-90f5-39b813174213 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:04.622691 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23066907-5ec2-4bb2-90f5-39b813174213 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:04.624555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-23066907-5ec2-4bb2-90f5-39b813174213 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:04.733605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23066907-5ec2-4bb2-90f5-39b813174213 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:04.735151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23066907-5ec2-4bb2-90f5-39b813174213 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:04.752897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 654/1314] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:07.193856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce311ba-299b-4ab0-a85b-269226f0fd33 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:07.194384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce311ba-299b-4ab0-a85b-269226f0fd33 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:07.194516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce311ba-299b-4ab0-a85b-269226f0fd33 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:07.195777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 661/1315] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:07.201786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f720df5-a412-4cd8-9c88-afa87641d436 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:07.202234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f720df5-a412-4cd8-9c88-afa87641d436 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:07.202329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f720df5-a412-4cd8-9c88-afa87641d436 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:07.268589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8f720df5-a412-4cd8-9c88-afa87641d436 None None] MFA Rules not processed for user `18674c9f5ca2436b921d8f52a40444be`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:07.269791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8f720df5-a412-4cd8-9c88-afa87641d436 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:07.321787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:07.369194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 655/1316] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:07 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:07.669918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e565e3-3f87-41b3-a740-403531b3c091 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:07.670205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e565e3-3f87-41b3-a740-403531b3c091 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:07.670484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e565e3-3f87-41b3-a740-403531b3c091 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:07.671806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 662/1317] 10.222.0.25 () {54 vars in 964 bytes} [Mon Sep 26 01:28:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:07.677581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217111fe-a391-4e69-823f-74044c19c72e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:07.677864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217111fe-a391-4e69-823f-74044c19c72e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:07.678139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217111fe-a391-4e69-823f-74044c19c72e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:07.755377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-217111fe-a391-4e69-823f-74044c19c72e None None] MFA Rules not processed for user `0478ebab37fb4f488eef84f7a241e206`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:07.756618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-217111fe-a391-4e69-823f-74044c19c72e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:07.831809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 656/1318] 10.222.0.25 () {58 vars in 1061 bytes} [Mon Sep 26 01:28:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:07.841001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e565e3-3f87-41b3-a740-403531b3c091 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:07.842553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f3e565e3-3f87-41b3-a740-403531b3c091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:07.852166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:07.944852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e565e3-3f87-41b3-a740-403531b3c091 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:08.018140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed538b98-85b0-416a-a1db-314edd6589b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:08.018737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed538b98-85b0-416a-a1db-314edd6589b1 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:08.019013 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed538b98-85b0-416a-a1db-314edd6589b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:08.019283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed538b98-85b0-416a-a1db-314edd6589b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:08.020903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ed538b98-85b0-416a-a1db-314edd6589b1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:08.083786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:08.131323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed538b98-85b0-416a-a1db-314edd6589b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:08.132798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed538b98-85b0-416a-a1db-314edd6589b1 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:08.151401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 663/1319] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:07 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:28:11.229270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a297471-fce3-4d98-b0d1-c91e37683257 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:11.229913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a297471-fce3-4d98-b0d1-c91e37683257 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:11.230025 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a297471-fce3-4d98-b0d1-c91e37683257 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:11.231297 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 657/1320] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:11.236685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd5f9cc-d045-41f3-9f58-31a6bdadd3a3 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:11.237132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd5f9cc-d045-41f3-9f58-31a6bdadd3a3 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:11.237242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd5f9cc-d045-41f3-9f58-31a6bdadd3a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:11.302023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7bd5f9cc-d045-41f3-9f58-31a6bdadd3a3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:11.303157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7bd5f9cc-d045-41f3-9f58-31a6bdadd3a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:11.399409 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 664/1321] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:11 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:11.689073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a297471-fce3-4d98-b0d1-c91e37683257 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:11.690610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8a297471-fce3-4d98-b0d1-c91e37683257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:11.743323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:11.796575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a297471-fce3-4d98-b0d1-c91e37683257 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:11.869059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df47637-41f0-4f06-9af5-fa45348cf62c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:11.869658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df47637-41f0-4f06-9af5-fa45348cf62c service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:11.869938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df47637-41f0-4f06-9af5-fa45348cf62c service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:11.870813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df47637-41f0-4f06-9af5-fa45348cf62c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:11.872491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3df47637-41f0-4f06-9af5-fa45348cf62c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:11.967212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:11.981583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df47637-41f0-4f06-9af5-fa45348cf62c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:11.983019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df47637-41f0-4f06-9af5-fa45348cf62c service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:12.000702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 658/1322] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:11 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:14.845376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb862ac-e480-4d10-aa7b-6e9f145ba375 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:14.845971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb862ac-e480-4d10-aa7b-6e9f145ba375 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:14.845971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb862ac-e480-4d10-aa7b-6e9f145ba375 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:14.847261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 665/1323] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:14.852641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855be431-7f56-4d1a-9643-622d80cd59d5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:14.853052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855be431-7f56-4d1a-9643-622d80cd59d5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:14.853187 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855be431-7f56-4d1a-9643-622d80cd59d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:14.930903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-855be431-7f56-4d1a-9643-622d80cd59d5 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:14.932144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-855be431-7f56-4d1a-9643-622d80cd59d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:15.037685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 659/1324] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:15.329910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb862ac-e480-4d10-aa7b-6e9f145ba375 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:15.331490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ddb862ac-e480-4d10-aa7b-6e9f145ba375 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:15.351516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:15.437184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb862ac-e480-4d10-aa7b-6e9f145ba375 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:15.510330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac33413-245f-4275-a241-69f41f1addba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:15.511473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac33413-245f-4275-a241-69f41f1addba service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:15.511753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac33413-245f-4275-a241-69f41f1addba service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:15.512006 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac33413-245f-4275-a241-69f41f1addba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:15.513640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fac33413-245f-4275-a241-69f41f1addba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:15.587259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:15.625322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac33413-245f-4275-a241-69f41f1addba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:15.626773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac33413-245f-4275-a241-69f41f1addba service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:15.645290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 666/1325] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:17.909862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8141dd9-7698-4c6a-b28d-a3a6feedaeba None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:17.910408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8141dd9-7698-4c6a-b28d-a3a6feedaeba None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:17.910408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8141dd9-7698-4c6a-b28d-a3a6feedaeba None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:17.911744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 660/1326] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:17.917398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e8a31-f18e-4bca-838b-383f1a7aeffc None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:17.917869 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e8a31-f18e-4bca-838b-383f1a7aeffc None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:17.918224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e8a31-f18e-4bca-838b-383f1a7aeffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:17.983497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-3b2e8a31-f18e-4bca-838b-383f1a7aeffc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:17.984723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3b2e8a31-f18e-4bca-838b-383f1a7aeffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:18.095516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 667/1327] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:28:18.377227 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8141dd9-7698-4c6a-b28d-a3a6feedaeba None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:18.378747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d8141dd9-7698-4c6a-b28d-a3a6feedaeba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:18.441697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:18.486303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8141dd9-7698-4c6a-b28d-a3a6feedaeba None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:18.561211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e13188-693c-40f9-b452-09dc0a596e47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:18.561814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e13188-693c-40f9-b452-09dc0a596e47 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:18.562100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e13188-693c-40f9-b452-09dc0a596e47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:18.562370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e13188-693c-40f9-b452-09dc0a596e47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:18.563978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d1e13188-693c-40f9-b452-09dc0a596e47 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:18.671619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:18.673910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e13188-693c-40f9-b452-09dc0a596e47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:18.675366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e13188-693c-40f9-b452-09dc0a596e47 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:18.694238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 661/1328] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:20.938584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e487bc0-73aa-486e-b4b0-7e381af41411 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:20.939097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e487bc0-73aa-486e-b4b0-7e381af41411 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:20.939097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e487bc0-73aa-486e-b4b0-7e381af41411 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:20.940403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 668/1329] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:20.945719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba8760-a970-4a9c-96d9-4470f298e01f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:20.946149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba8760-a970-4a9c-96d9-4470f298e01f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:20.946280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba8760-a970-4a9c-96d9-4470f298e01f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:21.016472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-31ba8760-a970-4a9c-96d9-4470f298e01f None None] MFA Rules not processed for user `18674c9f5ca2436b921d8f52a40444be`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:21.017616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-31ba8760-a970-4a9c-96d9-4470f298e01f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:21.115502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 662/1330] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:20 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:21.398393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e487bc0-73aa-486e-b4b0-7e381af41411 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:21.399935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6e487bc0-73aa-486e-b4b0-7e381af41411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:21.433990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:21.510878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e487bc0-73aa-486e-b4b0-7e381af41411 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:21.586304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770ebff6-5070-4628-bd48-de143d405ba8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:21.586818 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ebff6-5070-4628-bd48-de143d405ba8 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:21.587093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ebff6-5070-4628-bd48-de143d405ba8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:21.587359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ebff6-5070-4628-bd48-de143d405ba8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:21.589548 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-770ebff6-5070-4628-bd48-de143d405ba8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:21.669187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:21.701295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770ebff6-5070-4628-bd48-de143d405ba8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:21.703225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770ebff6-5070-4628-bd48-de143d405ba8 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:21.721795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 669/1331] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:21 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:24.146596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72e1993-5b89-47f3-a8a6-cbccd632c7a5 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:24.147090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72e1993-5b89-47f3-a8a6-cbccd632c7a5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:24.147090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72e1993-5b89-47f3-a8a6-cbccd632c7a5 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:24.148412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 663/1332] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:24.154173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79be249-6b1a-42f8-9a53-ad2bc0ef5b7c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:24.154556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79be249-6b1a-42f8-9a53-ad2bc0ef5b7c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:24.154762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79be249-6b1a-42f8-9a53-ad2bc0ef5b7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:24.221415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e79be249-6b1a-42f8-9a53-ad2bc0ef5b7c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:24.222612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e79be249-6b1a-42f8-9a53-ad2bc0ef5b7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:24.317541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 670/1333] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:24.595103 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72e1993-5b89-47f3-a8a6-cbccd632c7a5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:24.596704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c72e1993-5b89-47f3-a8a6-cbccd632c7a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:24.664759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:24.698246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72e1993-5b89-47f3-a8a6-cbccd632c7a5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:24.771889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf31b4e-d57a-4964-a001-97f24f0fed04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:24.772513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf31b4e-d57a-4964-a001-97f24f0fed04 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:24.772792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf31b4e-d57a-4964-a001-97f24f0fed04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:24.773064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf31b4e-d57a-4964-a001-97f24f0fed04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:24.774640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cbf31b4e-d57a-4964-a001-97f24f0fed04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:24.882874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf31b4e-d57a-4964-a001-97f24f0fed04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:24.884230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf31b4e-d57a-4964-a001-97f24f0fed04 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:24.894600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:24.903390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 664/1334] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:27.099030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ae8a53-5398-4518-b76b-11aa1e74733e None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:27.099505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ae8a53-5398-4518-b76b-11aa1e74733e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:27.099505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ae8a53-5398-4518-b76b-11aa1e74733e None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:27.100897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 671/1335] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:27.106420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9ffebe-01e6-4b29-8fe6-fa8769aa5654 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:27.106845 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9ffebe-01e6-4b29-8fe6-fa8769aa5654 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:27.106962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9ffebe-01e6-4b29-8fe6-fa8769aa5654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:27.173218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1f9ffebe-01e6-4b29-8fe6-fa8769aa5654 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:27.174412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1f9ffebe-01e6-4b29-8fe6-fa8769aa5654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:27.269907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 665/1336] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:27.553452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ae8a53-5398-4518-b76b-11aa1e74733e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:27.555023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a9ae8a53-5398-4518-b76b-11aa1e74733e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:27.598230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:27.662561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ae8a53-5398-4518-b76b-11aa1e74733e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:27.736547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7448c3bb-2abe-4602-8317-d8a533adf12d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:27.737147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448c3bb-2abe-4602-8317-d8a533adf12d service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:27.737422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448c3bb-2abe-4602-8317-d8a533adf12d service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:27.737696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7448c3bb-2abe-4602-8317-d8a533adf12d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:27.739293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7448c3bb-2abe-4602-8317-d8a533adf12d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:27.821920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:27.848657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7448c3bb-2abe-4602-8317-d8a533adf12d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:27.850083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7448c3bb-2abe-4602-8317-d8a533adf12d service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:27.867511 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 672/1337] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:28:30.406781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bf3d23-ae9e-471b-86fa-889fad3fdf68 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:30.407346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bf3d23-ae9e-471b-86fa-889fad3fdf68 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:30.407346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bf3d23-ae9e-471b-86fa-889fad3fdf68 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:30.408727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 666/1338] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:30.413964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e678964-cffc-41ba-a5b0-a01ee10d2c55 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:30.414364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e678964-cffc-41ba-a5b0-a01ee10d2c55 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:30.414517 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e678964-cffc-41ba-a5b0-a01ee10d2c55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:30.479703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-5e678964-cffc-41ba-a5b0-a01ee10d2c55 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:30.480949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5e678964-cffc-41ba-a5b0-a01ee10d2c55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:30.576451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 673/1339] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:30 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:30.858722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bf3d23-ae9e-471b-86fa-889fad3fdf68 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:30.860264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-52bf3d23-ae9e-471b-86fa-889fad3fdf68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:30.936378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:30.964537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bf3d23-ae9e-471b-86fa-889fad3fdf68 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:31.036597 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9703c0ff-0de4-45c7-a1b9-dfdceecf9715 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:31.037193 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9703c0ff-0de4-45c7-a1b9-dfdceecf9715 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:31.037467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9703c0ff-0de4-45c7-a1b9-dfdceecf9715 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:31.037735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9703c0ff-0de4-45c7-a1b9-dfdceecf9715 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:31.039329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9703c0ff-0de4-45c7-a1b9-dfdceecf9715 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:31.151590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9703c0ff-0de4-45c7-a1b9-dfdceecf9715 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:31.152964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9703c0ff-0de4-45c7-a1b9-dfdceecf9715 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:31.170972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 667/1340] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:30 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:35.461648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3895eab-b8e5-464a-98b9-3e128eda2326 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:35.462150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3895eab-b8e5-464a-98b9-3e128eda2326 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:35.462498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3895eab-b8e5-464a-98b9-3e128eda2326 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:35.464743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 674/1341] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:35.470719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db1d58-070a-479f-9ea4-b660787227f4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:35.470719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db1d58-070a-479f-9ea4-b660787227f4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:35.471271 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db1d58-070a-479f-9ea4-b660787227f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:35.476527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:35.544030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d9db1d58-070a-479f-9ea4-b660787227f4 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:35.545281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d9db1d58-070a-479f-9ea4-b660787227f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:35.659309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 668/1342] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:35.947971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3895eab-b8e5-464a-98b9-3e128eda2326 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:35.949547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d3895eab-b8e5-464a-98b9-3e128eda2326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:35.999570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:36.058272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3895eab-b8e5-464a-98b9-3e128eda2326 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:36.132549 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9aa7f7-427b-4135-b4c1-9e210eb703fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:36.133159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9aa7f7-427b-4135-b4c1-9e210eb703fb service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:36.133438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9aa7f7-427b-4135-b4c1-9e210eb703fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:36.133712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9aa7f7-427b-4135-b4c1-9e210eb703fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:36.135957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7c9aa7f7-427b-4135-b4c1-9e210eb703fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:36.230623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:36.247505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9aa7f7-427b-4135-b4c1-9e210eb703fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:36.248969 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9aa7f7-427b-4135-b4c1-9e210eb703fb service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:36.267012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 675/1343] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:39.060060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795f13e2-fae9-41a7-9cc5-785718e6a57f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:39.060570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795f13e2-fae9-41a7-9cc5-785718e6a57f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:39.060673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795f13e2-fae9-41a7-9cc5-785718e6a57f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:39.062157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 669/1344] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:39.067197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a178697-5105-4769-bc5f-f136036c1b6a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:39.067581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a178697-5105-4769-bc5f-f136036c1b6a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:39.067762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a178697-5105-4769-bc5f-f136036c1b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:39.134973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7a178697-5105-4769-bc5f-f136036c1b6a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:39.136201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7a178697-5105-4769-bc5f-f136036c1b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:39.233547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 676/1345] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:39 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:39.517257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795f13e2-fae9-41a7-9cc5-785718e6a57f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:39.518789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-795f13e2-fae9-41a7-9cc5-785718e6a57f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:39.615972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:39.636018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795f13e2-fae9-41a7-9cc5-785718e6a57f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:39.714310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b9aaa5-bd8b-4077-b598-84930c28fca6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:39.714959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b9aaa5-bd8b-4077-b598-84930c28fca6 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:39.715269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b9aaa5-bd8b-4077-b598-84930c28fca6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:39.715582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b9aaa5-bd8b-4077-b598-84930c28fca6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:39.717461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e2b9aaa5-bd8b-4077-b598-84930c28fca6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:39.832234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b9aaa5-bd8b-4077-b598-84930c28fca6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:39.833716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b9aaa5-bd8b-4077-b598-84930c28fca6 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:39.852516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 670/1346] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:39 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:43.972852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90db6ee3-7654-41eb-b53a-8123ee3527ab None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:43.973340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90db6ee3-7654-41eb-b53a-8123ee3527ab None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:43.973423 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90db6ee3-7654-41eb-b53a-8123ee3527ab None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:43.974741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 677/1347] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:43.980598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1660c2-8965-4943-b108-29d6bda8eb4f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:43.981369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1660c2-8965-4943-b108-29d6bda8eb4f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:43.981369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1660c2-8965-4943-b108-29d6bda8eb4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:43.997386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:44.059321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3d1660c2-8965-4943-b108-29d6bda8eb4f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:44.060634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3d1660c2-8965-4943-b108-29d6bda8eb4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:44.177714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 671/1348] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:44.465310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90db6ee3-7654-41eb-b53a-8123ee3527ab None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:28:44.466881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-90db6ee3-7654-41eb-b53a-8123ee3527ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:44.525487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:44.570084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90db6ee3-7654-41eb-b53a-8123ee3527ab None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:28:44.642894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48252b2-3663-47af-95e3-7bf4a95a2d38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:28:44.643494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48252b2-3663-47af-95e3-7bf4a95a2d38 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:44.643773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48252b2-3663-47af-95e3-7bf4a95a2d38 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:44.644048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48252b2-3663-47af-95e3-7bf4a95a2d38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:44.645673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f48252b2-3663-47af-95e3-7bf4a95a2d38 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:44.750571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:44.754733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48252b2-3663-47af-95e3-7bf4a95a2d38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:28:44.756170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48252b2-3663-47af-95e3-7bf4a95a2d38 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:28:44.774275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 678/1349] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:28:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:28:49.755973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2bbc3b-950c-4da0-8201-af64db190f5c None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:49.756912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2bbc3b-950c-4da0-8201-af64db190f5c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:49.757400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2bbc3b-950c-4da0-8201-af64db190f5c None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:49.760046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 672/1350] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:28:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:49.768387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ebc672-6524-4615-9da6-a8f5ddfdcbb5 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:49.768387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ebc672-6524-4615-9da6-a8f5ddfdcbb5 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:49.768387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ebc672-6524-4615-9da6-a8f5ddfdcbb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:49.893989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-28ebc672-6524-4615-9da6-a8f5ddfdcbb5 None None] MFA Rules not processed for user `427016f8da514dc4bbed21de08207dbb`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:49.895639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-28ebc672-6524-4615-9da6-a8f5ddfdcbb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:50.033039 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 679/1351] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:28:49 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:54.097419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff50f0b-c1bf-4cc0-a575-4a93984b83a2 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:54.098300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff50f0b-c1bf-4cc0-a575-4a93984b83a2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:54.098578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff50f0b-c1bf-4cc0-a575-4a93984b83a2 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:54.099928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 673/1352] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:28:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:54.105270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999bf26b-ecc9-4761-b77a-15c8d7140d1d None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:54.105697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999bf26b-ecc9-4761-b77a-15c8d7140d1d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:54.105792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999bf26b-ecc9-4761-b77a-15c8d7140d1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:54.163237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:54.176058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=59 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:54.189717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-999bf26b-ecc9-4761-b77a-15c8d7140d1d None None] MFA Rules not processed for user `427016f8da514dc4bbed21de08207dbb`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:54.190922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-999bf26b-ecc9-4761-b77a-15c8d7140d1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:54.302068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 680/1353] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:28:54 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:55.794745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a94a3d-d676-4e43-9e9c-49ab9ce4c116 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:55.795318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a94a3d-d676-4e43-9e9c-49ab9ce4c116 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:55.795318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a94a3d-d676-4e43-9e9c-49ab9ce4c116 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:55.797367 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 674/1354] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:28:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:55.802389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c61c611-1261-4f8f-8568-a904b023033a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:55.802808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c61c611-1261-4f8f-8568-a904b023033a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:55.803098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c61c611-1261-4f8f-8568-a904b023033a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:55.881937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-9c61c611-1261-4f8f-8568-a904b023033a None None] MFA Rules not processed for user `427016f8da514dc4bbed21de08207dbb`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:55.883699 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9c61c611-1261-4f8f-8568-a904b023033a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:55.925899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:28:55.991038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 681/1355] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:28:55 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:28:59.554124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67615124-c7bc-436d-9835-cc26860de67a None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:59.554715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67615124-c7bc-436d-9835-cc26860de67a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:59.554715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67615124-c7bc-436d-9835-cc26860de67a None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:59.558849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 675/1356] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:28:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:28:59.564293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b721f9f-7f3c-4a56-8852-03bce774734b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:28:59.564764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b721f9f-7f3c-4a56-8852-03bce774734b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:28:59.564943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b721f9f-7f3c-4a56-8852-03bce774734b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:28:59.641790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-8b721f9f-7f3c-4a56-8852-03bce774734b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:28:59.643127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8b721f9f-7f3c-4a56-8852-03bce774734b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:28:59.767293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 682/1357] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:28:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:00.081869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e9eafe-62cb-44e2-b1a1-bc790086a76d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:00.082577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e9eafe-62cb-44e2-b1a1-bc790086a76d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:00.083345 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e9eafe-62cb-44e2-b1a1-bc790086a76d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:00.085302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 676/1358] 10.222.0.25 () {54 vars in 961 bytes} [Mon Sep 26 01:29:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:00.093566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4886e5c2-bf66-4dc0-b733-1519c701b79b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:00.094041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4886e5c2-bf66-4dc0-b733-1519c701b79b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:00.094643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4886e5c2-bf66-4dc0-b733-1519c701b79b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:00.116163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:00.182594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-4886e5c2-bf66-4dc0-b733-1519c701b79b None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:00.184322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4886e5c2-bf66-4dc0-b733-1519c701b79b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:00.326544 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 683/1359] 10.222.0.25 () {58 vars in 1058 bytes} [Mon Sep 26 01:29:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:00.334932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e9eafe-62cb-44e2-b1a1-bc790086a76d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:00.337282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c1e9eafe-62cb-44e2-b1a1-bc790086a76d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:00.499067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e9eafe-62cb-44e2-b1a1-bc790086a76d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:00.579538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:00.623949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f02c25-5a2f-483b-ad0a-3d7325b9bcfb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:00.624636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f02c25-5a2f-483b-ad0a-3d7325b9bcfb service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:00.624918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f02c25-5a2f-483b-ad0a-3d7325b9bcfb service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:00.625194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f02c25-5a2f-483b-ad0a-3d7325b9bcfb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:00.626851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-45f02c25-5a2f-483b-ad0a-3d7325b9bcfb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:00.742878 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f02c25-5a2f-483b-ad0a-3d7325b9bcfb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:00.744584 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f02c25-5a2f-483b-ad0a-3d7325b9bcfb service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:00.765160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 677/1360] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:03.166713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7ca6a0-4f78-44e6-8e4e-d4a2ba32e280 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:03.166713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7ca6a0-4f78-44e6-8e4e-d4a2ba32e280 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:03.167516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7ca6a0-4f78-44e6-8e4e-d4a2ba32e280 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:03.171720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 684/1361] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:03 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:03.176041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff325453-87f0-4091-be24-89312da861b5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:03.176642 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff325453-87f0-4091-be24-89312da861b5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:03.176864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff325453-87f0-4091-be24-89312da861b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:03.255241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ff325453-87f0-4091-be24-89312da861b5 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:03.256694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ff325453-87f0-4091-be24-89312da861b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:03.302584 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:03.368537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 678/1362] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:03.662375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7ca6a0-4f78-44e6-8e4e-d4a2ba32e280 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:03.665638 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ba7ca6a0-4f78-44e6-8e4e-d4a2ba32e280 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:03.746844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:03.859262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7ca6a0-4f78-44e6-8e4e-d4a2ba32e280 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:03.995590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e05f03-17b2-40a2-8a17-e7492e242405 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:03.996126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e05f03-17b2-40a2-8a17-e7492e242405 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:03.996437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e05f03-17b2-40a2-8a17-e7492e242405 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:03.996714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e05f03-17b2-40a2-8a17-e7492e242405 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:03.998343 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-77e05f03-17b2-40a2-8a17-e7492e242405 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:04.021329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:04.092709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e05f03-17b2-40a2-8a17-e7492e242405 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:04.094427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e05f03-17b2-40a2-8a17-e7492e242405 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:04.113336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 685/1363] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:06.628208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dab33b-036a-4fb9-b6cb-da92c57cec81 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:06.628765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dab33b-036a-4fb9-b6cb-da92c57cec81 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:06.628765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dab33b-036a-4fb9-b6cb-da92c57cec81 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:06.630031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 679/1364] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:06.635766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a0d0c3-75f4-4efb-9202-346916af200a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:06.636543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a0d0c3-75f4-4efb-9202-346916af200a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:06.636823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a0d0c3-75f4-4efb-9202-346916af200a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:06.708987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b7a0d0c3-75f4-4efb-9202-346916af200a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:06.711012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b7a0d0c3-75f4-4efb-9202-346916af200a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:06.814804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:06.817636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 686/1365] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:07.117372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59046ab2-6885-48a3-a224-1100c3001459 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:07.117796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59046ab2-6885-48a3-a224-1100c3001459 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:07.118210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59046ab2-6885-48a3-a224-1100c3001459 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:07.119652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 680/1366] 10.222.0.25 () {54 vars in 961 bytes} [Mon Sep 26 01:29:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:07.125076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8ce43-114f-412b-b9c3-dc53412280eb None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:07.125462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8ce43-114f-412b-b9c3-dc53412280eb None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:07.125839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8ce43-114f-412b-b9c3-dc53412280eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:07.207470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-57e8ce43-114f-412b-b9c3-dc53412280eb None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:07.208891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-57e8ce43-114f-412b-b9c3-dc53412280eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:07.333632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 687/1367] 10.222.0.25 () {58 vars in 1058 bytes} [Mon Sep 26 01:29:07 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:07.345954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59046ab2-6885-48a3-a224-1100c3001459 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:07.347508 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-59046ab2-6885-48a3-a224-1100c3001459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:07.506864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59046ab2-6885-48a3-a224-1100c3001459 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:07.530469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:07.631341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516dd261-b6a1-4ff2-a7b9-400fac27e4e3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:07.631964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516dd261-b6a1-4ff2-a7b9-400fac27e4e3 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:07.632212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516dd261-b6a1-4ff2-a7b9-400fac27e4e3 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:07.632636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516dd261-b6a1-4ff2-a7b9-400fac27e4e3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:07.634350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-516dd261-b6a1-4ff2-a7b9-400fac27e4e3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:07.739057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516dd261-b6a1-4ff2-a7b9-400fac27e4e3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:07.740703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516dd261-b6a1-4ff2-a7b9-400fac27e4e3 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:07.761061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 681/1368] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:07.777712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbb7f87-97ab-481f-a887-cf629bab7384 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:07.778184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbb7f87-97ab-481f-a887-cf629bab7384 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:07.778265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbb7f87-97ab-481f-a887-cf629bab7384 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:07.779734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 688/1369] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:07.784737 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09984ced-5f33-4b78-b129-b478c4b908f6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:07.785030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09984ced-5f33-4b78-b129-b478c4b908f6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:07.785302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09984ced-5f33-4b78-b129-b478c4b908f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:07.822509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:07.862743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-09984ced-5f33-4b78-b129-b478c4b908f6 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:07.864147 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-09984ced-5f33-4b78-b129-b478c4b908f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:08.001067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 682/1370] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:08.297711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834d42ab-9976-491e-b4c3-08ab2a6fa84b None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:08.297997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834d42ab-9976-491e-b4c3-08ab2a6fa84b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:08.298271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834d42ab-9976-491e-b4c3-08ab2a6fa84b None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:08.299856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 689/1371] 10.222.0.25 () {54 vars in 963 bytes} [Mon Sep 26 01:29:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:08.305664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec16bb9-ba2f-4394-a078-2a00661b63ea None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:08.306064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec16bb9-ba2f-4394-a078-2a00661b63ea None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:08.306427 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec16bb9-ba2f-4394-a078-2a00661b63ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:08.385912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1ec16bb9-ba2f-4394-a078-2a00661b63ea None None] MFA Rules not processed for user `2bb5f76eb08741f3a8c7a3a140e01e24`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:08.387262 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1ec16bb9-ba2f-4394-a078-2a00661b63ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:08.437066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:08.474117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 683/1372] 10.222.0.25 () {58 vars in 1060 bytes} [Mon Sep 26 01:29:08 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:08.482494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834d42ab-9976-491e-b4c3-08ab2a6fa84b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:08.484068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-834d42ab-9976-491e-b4c3-08ab2a6fa84b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:08.541161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:08.602435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834d42ab-9976-491e-b4c3-08ab2a6fa84b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:08.685709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b4fe84-6672-49b9-90da-41df36550a83 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:08.686576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b4fe84-6672-49b9-90da-41df36550a83 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:08.686955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b4fe84-6672-49b9-90da-41df36550a83 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:08.687327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b4fe84-6672-49b9-90da-41df36550a83 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:08.689067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-78b4fe84-6672-49b9-90da-41df36550a83 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:08.799924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:08.815783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b4fe84-6672-49b9-90da-41df36550a83 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:08.817255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b4fe84-6672-49b9-90da-41df36550a83 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:08.840054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 690/1373] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:10.057753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dc25ab-43d8-4cac-9510-7a8ea3ebd0cc None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:10.058218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dc25ab-43d8-4cac-9510-7a8ea3ebd0cc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:10.058331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dc25ab-43d8-4cac-9510-7a8ea3ebd0cc None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:10.059671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 684/1374] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:10.065003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a514d5f-5c82-4576-9df4-39aaf0fc8ddd None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:10.065403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a514d5f-5c82-4576-9df4-39aaf0fc8ddd None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:10.065560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a514d5f-5c82-4576-9df4-39aaf0fc8ddd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:10.133029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-1a514d5f-5c82-4576-9df4-39aaf0fc8ddd None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:10.134173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1a514d5f-5c82-4576-9df4-39aaf0fc8ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:10.231533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 691/1375] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:10.524935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dc25ab-43d8-4cac-9510-7a8ea3ebd0cc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:10.526512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a4dc25ab-43d8-4cac-9510-7a8ea3ebd0cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:10.687400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dc25ab-43d8-4cac-9510-7a8ea3ebd0cc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:10.742945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:10.807992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bfee89-781e-4a22-bef2-469dee4e6687 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:10.808704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfee89-781e-4a22-bef2-469dee4e6687 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:10.808990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfee89-781e-4a22-bef2-469dee4e6687 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:10.809263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfee89-781e-4a22-bef2-469dee4e6687 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:10.810918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-94bfee89-781e-4a22-bef2-469dee4e6687 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:10.907625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bfee89-781e-4a22-bef2-469dee4e6687 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:10.909261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bfee89-781e-4a22-bef2-469dee4e6687 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:10.928439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 685/1376] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:13.176004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184338b-68af-464e-9322-7fc51c6b89ea None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:13.176557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184338b-68af-464e-9322-7fc51c6b89ea None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:13.176665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184338b-68af-464e-9322-7fc51c6b89ea None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:13.178074 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 692/1377] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:13.183260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e04f9f-ddba-4437-8e4d-fbab29909c49 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:13.183679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e04f9f-ddba-4437-8e4d-fbab29909c49 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:13.183852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e04f9f-ddba-4437-8e4d-fbab29909c49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:13.258523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-24e04f9f-ddba-4437-8e4d-fbab29909c49 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:13.259804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-24e04f9f-ddba-4437-8e4d-fbab29909c49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:13.267764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:13.368579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 686/1378] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:13.663227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6184338b-68af-464e-9322-7fc51c6b89ea None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:13.665322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6184338b-68af-464e-9322-7fc51c6b89ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:13.850172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:13.946415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6184338b-68af-464e-9322-7fc51c6b89ea None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:14.076844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6121c7a4-0179-4cd0-9aa5-ed0ae48bcc46 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:14.077444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6121c7a4-0179-4cd0-9aa5-ed0ae48bcc46 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:14.077716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6121c7a4-0179-4cd0-9aa5-ed0ae48bcc46 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:14.077980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6121c7a4-0179-4cd0-9aa5-ed0ae48bcc46 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:14.079906 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6121c7a4-0179-4cd0-9aa5-ed0ae48bcc46 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:14.102504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:14.184932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6121c7a4-0179-4cd0-9aa5-ed0ae48bcc46 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:14.186630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6121c7a4-0179-4cd0-9aa5-ed0ae48bcc46 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:14.207741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 693/1379] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 545 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:16.460336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d370db-1297-4c1c-836c-19fd8be23d2e None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:16.460336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d370db-1297-4c1c-836c-19fd8be23d2e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:16.460336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d370db-1297-4c1c-836c-19fd8be23d2e None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:16.460336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 687/1380] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:16.467613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aabbd4b-602e-47f1-a977-d9b6e767b056 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:16.467613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aabbd4b-602e-47f1-a977-d9b6e767b056 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:16.467613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aabbd4b-602e-47f1-a977-d9b6e767b056 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:16.547825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0aabbd4b-602e-47f1-a977-d9b6e767b056 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:16.549117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0aabbd4b-602e-47f1-a977-d9b6e767b056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:16.642362 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:16.648962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 694/1381] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:16.986436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d370db-1297-4c1c-836c-19fd8be23d2e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:16.987972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-28d370db-1297-4c1c-836c-19fd8be23d2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:17.139902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:17.152153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d370db-1297-4c1c-836c-19fd8be23d2e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:17.278539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd08aa4-15be-4e0a-a3f4-47c580638188 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:17.279065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd08aa4-15be-4e0a-a3f4-47c580638188 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:17.279349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd08aa4-15be-4e0a-a3f4-47c580638188 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:17.279619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd08aa4-15be-4e0a-a3f4-47c580638188 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:17.281257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0cd08aa4-15be-4e0a-a3f4-47c580638188 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:17.374026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd08aa4-15be-4e0a-a3f4-47c580638188 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:17.375656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd08aa4-15be-4e0a-a3f4-47c580638188 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:17.386247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:17.397256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 688/1382] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:19.730984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00be401-e222-4747-b5b8-cad3fcfbb7f8 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:19.731515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00be401-e222-4747-b5b8-cad3fcfbb7f8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:19.731515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00be401-e222-4747-b5b8-cad3fcfbb7f8 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:19.733114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 695/1383] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:19.738501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fbce1b-363c-4ebd-b302-6c8b4102abfe None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:19.738896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fbce1b-363c-4ebd-b302-6c8b4102abfe None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:19.739047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fbce1b-363c-4ebd-b302-6c8b4102abfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:19.812664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-76fbce1b-363c-4ebd-b302-6c8b4102abfe None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:19.813984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-76fbce1b-363c-4ebd-b302-6c8b4102abfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:19.917839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 689/1384] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:20.206305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00be401-e222-4747-b5b8-cad3fcfbb7f8 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:20.208091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c00be401-e222-4747-b5b8-cad3fcfbb7f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:20.373365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00be401-e222-4747-b5b8-cad3fcfbb7f8 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:20.449318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:20.502304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cc5dab-1503-4531-99db-34b4160b29bd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:20.502904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cc5dab-1503-4531-99db-34b4160b29bd service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:20.503221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cc5dab-1503-4531-99db-34b4160b29bd service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:20.503522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cc5dab-1503-4531-99db-34b4160b29bd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:20.505451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-16cc5dab-1503-4531-99db-34b4160b29bd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:20.606852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cc5dab-1503-4531-99db-34b4160b29bd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:20.608510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cc5dab-1503-4531-99db-34b4160b29bd service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:20.627384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 696/1385] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:23.527778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6365dc61-424e-4dbd-98b0-b9167eb6d827 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:23.528431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6365dc61-424e-4dbd-98b0-b9167eb6d827 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:23.528568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6365dc61-424e-4dbd-98b0-b9167eb6d827 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:23.530460 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 690/1386] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:23.536266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda3da35-8d17-44d0-87c2-42456ee26a15 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:23.536715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda3da35-8d17-44d0-87c2-42456ee26a15 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:23.537005 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda3da35-8d17-44d0-87c2-42456ee26a15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:23.621121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-cda3da35-8d17-44d0-87c2-42456ee26a15 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:23.622691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-cda3da35-8d17-44d0-87c2-42456ee26a15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:23.659605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:23.733779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 697/1387] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:24.027872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6365dc61-424e-4dbd-98b0-b9167eb6d827 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:24.029920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6365dc61-424e-4dbd-98b0-b9167eb6d827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:24.129275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:24.203652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6365dc61-424e-4dbd-98b0-b9167eb6d827 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:24.335832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f042f435-330b-43e7-b2aa-3bb16b439488 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:24.336821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f042f435-330b-43e7-b2aa-3bb16b439488 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:24.337180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f042f435-330b-43e7-b2aa-3bb16b439488 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:24.337577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f042f435-330b-43e7-b2aa-3bb16b439488 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:24.339840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f042f435-330b-43e7-b2aa-3bb16b439488 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:24.395401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:24.453126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f042f435-330b-43e7-b2aa-3bb16b439488 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:24.454734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f042f435-330b-43e7-b2aa-3bb16b439488 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:24.473534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 691/1388] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:26.682160 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4016dde-2d75-4ce9-ac75-931d5fda54d4 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:26.682733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4016dde-2d75-4ce9-ac75-931d5fda54d4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:26.682733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4016dde-2d75-4ce9-ac75-931d5fda54d4 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:26.684061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 698/1389] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:26.689461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f180e30e-c8e3-4c29-a932-e92fea50261a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:26.689877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f180e30e-c8e3-4c29-a932-e92fea50261a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:26.690061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f180e30e-c8e3-4c29-a932-e92fea50261a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:26.700070 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be91920-2567-43fa-a3b8-2b87a9913cd4 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:26.700372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be91920-2567-43fa-a3b8-2b87a9913cd4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:26.701056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be91920-2567-43fa-a3b8-2b87a9913cd4 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:26.702201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 699/1390] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:29:26.707169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f518c6d-8049-4e70-b603-efdbf75d624e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:26.707454 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f518c6d-8049-4e70-b603-efdbf75d624e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:26.707726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f518c6d-8049-4e70-b603-efdbf75d624e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:26.759467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f180e30e-c8e3-4c29-a932-e92fea50261a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:26.760758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f180e30e-c8e3-4c29-a932-e92fea50261a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:26.776849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-4f518c6d-8049-4e70-b603-efdbf75d624e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:26.778049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4f518c6d-8049-4e70-b603-efdbf75d624e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:26.868389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 692/1391] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:26.878444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 700/1392] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:29:27.154475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f180e30e-c8e3-4c29-a932-e92fea50261a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:27.155956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f180e30e-c8e3-4c29-a932-e92fea50261a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:27.171366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:27.184009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad72d84-d3a8-4b0f-ba6f-1d297706e215 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:27.184290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad72d84-d3a8-4b0f-ba6f-1d297706e215 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:27.184592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad72d84-d3a8-4b0f-ba6f-1d297706e215 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:27.185945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 701/1393] 10.222.0.25 () {54 vars in 962 bytes} [Mon Sep 26 01:29:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:29:27.191534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e967ee-4a96-489b-a995-f873a7ac8e93 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:27.191823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e967ee-4a96-489b-a995-f873a7ac8e93 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:27.192247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e967ee-4a96-489b-a995-f873a7ac8e93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:27.209017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:27.270828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-26e967ee-4a96-489b-a995-f873a7ac8e93 None None] MFA Rules not processed for user `e07b6890d3004dd78faed48a1ab90086`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:27.272045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-26e967ee-4a96-489b-a995-f873a7ac8e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:27.276872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f180e30e-c8e3-4c29-a932-e92fea50261a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:27.350440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 702/1394] 10.222.0.25 () {58 vars in 1059 bytes} [Mon Sep 26 01:29:27 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:29:27.359744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e967ee-4a96-489b-a995-f873a7ac8e93 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:27.367669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e967ee-4a96-489b-a995-f873a7ac8e93 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:27.393138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaea087e-bc4b-4cde-8c11-44b75124d2ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:27.393748 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea087e-bc4b-4cde-8c11-44b75124d2ca service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:27.394028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea087e-bc4b-4cde-8c11-44b75124d2ca service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:27.394693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaea087e-bc4b-4cde-8c11-44b75124d2ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:27.395912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-eaea087e-bc4b-4cde-8c11-44b75124d2ca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:27.405464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:27.443392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea01274a-077e-4a67-9080-05783a5c2ec7 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e07b6890d3004dd78faed48a1ab90086', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:27.443999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea01274a-077e-4a67-9080-05783a5c2ec7 service glance] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:27.444282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea01274a-077e-4a67-9080-05783a5c2ec7 service glance] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:27.445208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea01274a-077e-4a67-9080-05783a5c2ec7 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:27.446831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ea01274a-077e-4a67-9080-05783a5c2ec7 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:27.479154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:27.491018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaea087e-bc4b-4cde-8c11-44b75124d2ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:27.492851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaea087e-bc4b-4cde-8c11-44b75124d2ca service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:27.511913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 693/1395] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:27.561195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea01274a-077e-4a67-9080-05783a5c2ec7 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:27.562625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea01274a-077e-4a67-9080-05783a5c2ec7 service glance] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:27.586632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 703/1396] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:29:28.339826 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27af2b5-79f2-4b0b-a5ae-6934e195cb55 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:28.340437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27af2b5-79f2-4b0b-a5ae-6934e195cb55 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:28.340551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27af2b5-79f2-4b0b-a5ae-6934e195cb55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:28.407796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a27af2b5-79f2-4b0b-a5ae-6934e195cb55 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:28.409050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a27af2b5-79f2-4b0b-a5ae-6934e195cb55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:28.496816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:28.500257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 694/1397] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:29:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:28.547626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a73b8c6-a5bc-4e35-a3ed-456893cec1e1 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:28.548088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a73b8c6-a5bc-4e35-a3ed-456893cec1e1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:28.548203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a73b8c6-a5bc-4e35-a3ed-456893cec1e1 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:28.549708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 704/1398] 10.222.0.25 () {54 vars in 943 bytes} [Mon Sep 26 01:29:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:28.555870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561b54a1-4b68-41f0-9b5d-42b8b05152eb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:28.556154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561b54a1-4b68-41f0-9b5d-42b8b05152eb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:28.556460 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561b54a1-4b68-41f0-9b5d-42b8b05152eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:28.636000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-561b54a1-4b68-41f0-9b5d-42b8b05152eb None None] MFA Rules not processed for user `e542b14838e2466fb59d17e9a20682dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:28.637201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-561b54a1-4b68-41f0-9b5d-42b8b05152eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:28.745580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 695/1399] 10.222.0.25 () {58 vars in 1040 bytes} [Mon Sep 26 01:29:28 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:28.755409 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a73b8c6-a5bc-4e35-a3ed-456893cec1e1 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:28.757679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5a73b8c6-a5bc-4e35-a3ed-456893cec1e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:28.894106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:28.896606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a73b8c6-a5bc-4e35-a3ed-456893cec1e1 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:29.003458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9d3ff6-e0f9-4d73-ae93-e80524861236 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:29.004269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9d3ff6-e0f9-4d73-ae93-e80524861236 service swift] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:29.004795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9d3ff6-e0f9-4d73-ae93-e80524861236 service swift] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:29.005282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9d3ff6-e0f9-4d73-ae93-e80524861236 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:29.008319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ac9d3ff6-e0f9-4d73-ae93-e80524861236 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:29.096872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9d3ff6-e0f9-4d73-ae93-e80524861236 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:29.098474 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9d3ff6-e0f9-4d73-ae93-e80524861236 service swift] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:29.107596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 705/1400] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:29:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:29:29.724109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f4220-7f45-44af-b44c-d22eb91bc439 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:29.724718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f4220-7f45-44af-b44c-d22eb91bc439 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:29.724718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f4220-7f45-44af-b44c-d22eb91bc439 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:29.726082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 696/1401] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:29.731596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03f6e7e-aaa2-4785-a288-a481b1cd0c93 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:29.732077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03f6e7e-aaa2-4785-a288-a481b1cd0c93 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:29.732195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03f6e7e-aaa2-4785-a288-a481b1cd0c93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:29.777350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:29.811405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e03f6e7e-aaa2-4785-a288-a481b1cd0c93 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:29.812677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e03f6e7e-aaa2-4785-a288-a481b1cd0c93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:29.930882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 706/1402] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:30.221954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156f4220-7f45-44af-b44c-d22eb91bc439 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:30.223760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-156f4220-7f45-44af-b44c-d22eb91bc439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:30.295009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:30.393445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156f4220-7f45-44af-b44c-d22eb91bc439 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:30.539048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954701cb-554e-41bd-ac71-6cdc0dc981d2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:30.539653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954701cb-554e-41bd-ac71-6cdc0dc981d2 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:30.539945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954701cb-554e-41bd-ac71-6cdc0dc981d2 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:30.540243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954701cb-554e-41bd-ac71-6cdc0dc981d2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:30.541846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-954701cb-554e-41bd-ac71-6cdc0dc981d2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:30.560994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:30.640132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954701cb-554e-41bd-ac71-6cdc0dc981d2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:30.642158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954701cb-554e-41bd-ac71-6cdc0dc981d2 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:30.665757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 697/1403] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:32.909023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb22df5-bb63-47e1-ab57-7bcc6f4c3fdc None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:32.909611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb22df5-bb63-47e1-ab57-7bcc6f4c3fdc None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:32.909611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb22df5-bb63-47e1-ab57-7bcc6f4c3fdc None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:32.915484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 707/1404] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:32 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:32.916915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccfa03a-bf52-442f-9efb-1522fb97d0ee None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:32.917352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccfa03a-bf52-442f-9efb-1522fb97d0ee None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:32.917495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccfa03a-bf52-442f-9efb-1522fb97d0ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:32.992849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9ccfa03a-bf52-442f-9efb-1522fb97d0ee None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:32.994114 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9ccfa03a-bf52-442f-9efb-1522fb97d0ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:33.100312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:33.107956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 698/1405] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:33.399822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb22df5-bb63-47e1-ab57-7bcc6f4c3fdc None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:33.401786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-dfb22df5-bb63-47e1-ab57-7bcc6f4c3fdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:33.524849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:33.577075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb22df5-bb63-47e1-ab57-7bcc6f4c3fdc None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:33.703973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e67225-38a7-4795-9208-943d02790894 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:33.704747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e67225-38a7-4795-9208-943d02790894 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:33.705085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e67225-38a7-4795-9208-943d02790894 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:33.705417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e67225-38a7-4795-9208-943d02790894 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:33.708462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d7e67225-38a7-4795-9208-943d02790894 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:33.836692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:33.860129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e67225-38a7-4795-9208-943d02790894 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:33.861777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e67225-38a7-4795-9208-943d02790894 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:33.881327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 708/1406] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:36.093760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42411778-7b5b-4942-b0e8-d619b085dee0 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:36.094356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42411778-7b5b-4942-b0e8-d619b085dee0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:36.094356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42411778-7b5b-4942-b0e8-d619b085dee0 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:36.095652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 699/1407] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:29:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:36.101616 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d466c86-6f0e-4511-8b73-cf60042c1c0e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:36.102103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d466c86-6f0e-4511-8b73-cf60042c1c0e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:36.102380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d466c86-6f0e-4511-8b73-cf60042c1c0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:36.173192 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0d466c86-6f0e-4511-8b73-cf60042c1c0e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:36.174457 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0d466c86-6f0e-4511-8b73-cf60042c1c0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:36.280646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 709/1408] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:29:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:36.574010 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42411778-7b5b-4942-b0e8-d619b085dee0 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:36.575553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-42411778-7b5b-4942-b0e8-d619b085dee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:36.744821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42411778-7b5b-4942-b0e8-d619b085dee0 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:36.822076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:36.873107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdcf197-27a4-45bc-8bd6-a5ce68117edf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:36.873855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdcf197-27a4-45bc-8bd6-a5ce68117edf service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:36.874194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdcf197-27a4-45bc-8bd6-a5ce68117edf service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:36.874532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdcf197-27a4-45bc-8bd6-a5ce68117edf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:36.876192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cbdcf197-27a4-45bc-8bd6-a5ce68117edf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:36.980639 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdcf197-27a4-45bc-8bd6-a5ce68117edf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:36.982264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdcf197-27a4-45bc-8bd6-a5ce68117edf service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:37.001363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 700/1409] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:46.279058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=79 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:46.999146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=64 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:52.242066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee95e99-2f0f-496a-8a24-2cb0771ac5f6 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:52.242656 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee95e99-2f0f-496a-8a24-2cb0771ac5f6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:52.242656 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee95e99-2f0f-496a-8a24-2cb0771ac5f6 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:52.243950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 710/1410] 10.222.0.25 () {54 vars in 918 bytes} [Mon Sep 26 01:29:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:52.249238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774a5fe-ef96-4e12-960c-f2315d857970 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:52.249665 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774a5fe-ef96-4e12-960c-f2315d857970 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:52.249768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774a5fe-ef96-4e12-960c-f2315d857970 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:52.327850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5774a5fe-ef96-4e12-960c-f2315d857970 None None] MFA Rules not processed for user `427016f8da514dc4bbed21de08207dbb`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:52.329172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5774a5fe-ef96-4e12-960c-f2315d857970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:52.434080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 701/1411] 10.222.0.25 () {58 vars in 1015 bytes} [Mon Sep 26 01:29:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:52.448851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c1af3-5e27-4eb7-97a7-586e06d7c3a4 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:52.449159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c1af3-5e27-4eb7-97a7-586e06d7c3a4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:52.449430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c1af3-5e27-4eb7-97a7-586e06d7c3a4 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:52.450762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 711/1412] 10.222.0.25 () {54 vars in 918 bytes} [Mon Sep 26 01:29:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:52.455739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec31e1b-1633-471e-b111-4c0389144f03 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:52.456027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec31e1b-1633-471e-b111-4c0389144f03 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:52.456307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec31e1b-1633-471e-b111-4c0389144f03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:52.528789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:52.531890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9ec31e1b-1633-471e-b111-4c0389144f03 None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:52.533702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9ec31e1b-1633-471e-b111-4c0389144f03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:52.649375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 702/1413] 10.222.0.25 () {58 vars in 1015 bytes} [Mon Sep 26 01:29:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:55.214301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b8c75-9041-41f5-acfb-35baa333fb88 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:55.214882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b8c75-9041-41f5-acfb-35baa333fb88 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:55.214882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0b8c75-9041-41f5-acfb-35baa333fb88 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:55.216176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 712/1414] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:29:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:55.221682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03026173-965c-40c2-b4fc-b2946724cc41 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:55.222093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03026173-965c-40c2-b4fc-b2946724cc41 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:55.222235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03026173-965c-40c2-b4fc-b2946724cc41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:55.288805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-03026173-965c-40c2-b4fc-b2946724cc41 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:55.290014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-03026173-965c-40c2-b4fc-b2946724cc41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:55.344156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:55.387996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 703/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:29:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:55.679654 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0b8c75-9041-41f5-acfb-35baa333fb88 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:55.681293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1d0b8c75-9041-41f5-acfb-35baa333fb88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:55.827765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0b8c75-9041-41f5-acfb-35baa333fb88 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:55.901625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:55.941333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d20bff8-462b-4700-b0f9-96cbe601a028 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:55.941870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d20bff8-462b-4700-b0f9-96cbe601a028 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:55.942151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d20bff8-462b-4700-b0f9-96cbe601a028 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:55.942425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d20bff8-462b-4700-b0f9-96cbe601a028 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:55.944567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2d20bff8-462b-4700-b0f9-96cbe601a028 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:56.035642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d20bff8-462b-4700-b0f9-96cbe601a028 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:56.037636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d20bff8-462b-4700-b0f9-96cbe601a028 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:56.055709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 713/1416] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:29:57.199836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ec3c65-e6f8-42fb-b7b0-8c6429705345 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:57.200410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ec3c65-e6f8-42fb-b7b0-8c6429705345 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:57.200410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ec3c65-e6f8-42fb-b7b0-8c6429705345 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:57.201701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 704/1417] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:29:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:57.208146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b6f3e8-a126-4a7d-ad3b-11570cab3c60 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:57.208616 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b6f3e8-a126-4a7d-ad3b-11570cab3c60 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:57.208727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b6f3e8-a126-4a7d-ad3b-11570cab3c60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:57.284788 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-15b6f3e8-a126-4a7d-ad3b-11570cab3c60 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:57.285990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-15b6f3e8-a126-4a7d-ad3b-11570cab3c60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:57.314537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:57.385903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 714/1418] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:29:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:57.535767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ec3c65-e6f8-42fb-b7b0-8c6429705345 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:57.537338 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f8ec3c65-e6f8-42fb-b7b0-8c6429705345 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:57.682485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ec3c65-e6f8-42fb-b7b0-8c6429705345 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:57.724361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:57.797670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff6cbb1-d951-45e8-a48e-827198f828a3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:57.798194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff6cbb1-d951-45e8-a48e-827198f828a3 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:57.798474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff6cbb1-d951-45e8-a48e-827198f828a3 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:57.798747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff6cbb1-d951-45e8-a48e-827198f828a3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:57.800426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1ff6cbb1-d951-45e8-a48e-827198f828a3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:57.890568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff6cbb1-d951-45e8-a48e-827198f828a3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:57.892024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff6cbb1-d951-45e8-a48e-827198f828a3 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:57.909501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 705/1419] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:29:59.043823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b98af45-f393-4da5-92c0-96d064afa148 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:59.044400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b98af45-f393-4da5-92c0-96d064afa148 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:59.044400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b98af45-f393-4da5-92c0-96d064afa148 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:59.045712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 715/1420] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:29:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:29:59.052041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae6e43b-a5bc-450e-a8c5-861cf732cee5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:59.052493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae6e43b-a5bc-450e-a8c5-861cf732cee5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:59.052632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae6e43b-a5bc-450e-a8c5-861cf732cee5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:59.107095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:59.129222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-eae6e43b-a5bc-450e-a8c5-861cf732cee5 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:29:59.130419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-eae6e43b-a5bc-450e-a8c5-861cf732cee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:59.229458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 706/1421] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:29:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:29:59.381706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b98af45-f393-4da5-92c0-96d064afa148 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:29:59.383203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7b98af45-f393-4da5-92c0-96d064afa148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:59.528330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b98af45-f393-4da5-92c0-96d064afa148 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:29:59.541103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:29:59.641291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ba4362-d5df-4c11-bd15-3325b5927fbf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:29:59.641831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ba4362-d5df-4c11-bd15-3325b5927fbf service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:29:59.642106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ba4362-d5df-4c11-bd15-3325b5927fbf service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:29:59.642372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ba4362-d5df-4c11-bd15-3325b5927fbf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:29:59.643948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-99ba4362-d5df-4c11-bd15-3325b5927fbf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:29:59.732649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ba4362-d5df-4c11-bd15-3325b5927fbf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:29:59.734237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ba4362-d5df-4c11-bd15-3325b5927fbf service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:29:59.751764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 716/1422] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:29:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:00.873410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50b51df-c180-4f31-9771-7aa4a7005f70 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:00.873946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50b51df-c180-4f31-9771-7aa4a7005f70 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:00.873946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50b51df-c180-4f31-9771-7aa4a7005f70 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:00.875286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 707/1423] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:30:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:00.882105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e900e12-c095-4238-9dd7-4f9f826ce73c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:00.882569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e900e12-c095-4238-9dd7-4f9f826ce73c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:00.882684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e900e12-c095-4238-9dd7-4f9f826ce73c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:00.901935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:00.958650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-9e900e12-c095-4238-9dd7-4f9f826ce73c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:00.959901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9e900e12-c095-4238-9dd7-4f9f826ce73c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:01.055787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 717/1424] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:30:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:01.208595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50b51df-c180-4f31-9771-7aa4a7005f70 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:01.210305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f50b51df-c180-4f31-9771-7aa4a7005f70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:01.319534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:01.355667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50b51df-c180-4f31-9771-7aa4a7005f70 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:01.471133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fc3fdd-3b77-4d5d-830a-9f279f79618b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:01.471650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fc3fdd-3b77-4d5d-830a-9f279f79618b service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:01.471928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fc3fdd-3b77-4d5d-830a-9f279f79618b service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:01.472199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fc3fdd-3b77-4d5d-830a-9f279f79618b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:01.473889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e6fc3fdd-3b77-4d5d-830a-9f279f79618b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:01.549884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:01.563549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fc3fdd-3b77-4d5d-830a-9f279f79618b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:01.565198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fc3fdd-3b77-4d5d-830a-9f279f79618b service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:01.583167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 708/1425] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:02.706978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3028df52-520d-4529-b680-061a85c12d9a None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:02.707520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3028df52-520d-4529-b680-061a85c12d9a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:02.707619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3028df52-520d-4529-b680-061a85c12d9a None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:02.708927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 718/1426] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:30:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:02.716066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aa63a9-86aa-47e8-a853-a8b6fa1cf82a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:02.716764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aa63a9-86aa-47e8-a853-a8b6fa1cf82a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:02.716764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73aa63a9-86aa-47e8-a853-a8b6fa1cf82a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:02.792548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-73aa63a9-86aa-47e8-a853-a8b6fa1cf82a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:02.793736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-73aa63a9-86aa-47e8-a853-a8b6fa1cf82a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:02.886998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 709/1427] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:30:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:03.037901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3028df52-520d-4529-b680-061a85c12d9a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:03.039451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3028df52-520d-4529-b680-061a85c12d9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:03.122360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:03.183647 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3028df52-520d-4529-b680-061a85c12d9a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:03.296684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272665f7-e13c-4446-9143-c6adc76fd54b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:03.297194 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272665f7-e13c-4446-9143-c6adc76fd54b service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:03.297467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272665f7-e13c-4446-9143-c6adc76fd54b service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:03.297735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272665f7-e13c-4446-9143-c6adc76fd54b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:03.299319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-272665f7-e13c-4446-9143-c6adc76fd54b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:03.348289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:03.389181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272665f7-e13c-4446-9143-c6adc76fd54b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:03.390758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272665f7-e13c-4446-9143-c6adc76fd54b service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:03.408747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 719/1428] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:04.469918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec5c221-9144-4c0e-b3b9-ab5e482f0e25 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:04.470461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec5c221-9144-4c0e-b3b9-ab5e482f0e25 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:04.470461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec5c221-9144-4c0e-b3b9-ab5e482f0e25 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:04.471814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 710/1429] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:30:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:04.478289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b855b94-cb81-477d-9b17-0484597cc20e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:04.478741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b855b94-cb81-477d-9b17-0484597cc20e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:04.478835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b855b94-cb81-477d-9b17-0484597cc20e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:04.558765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-2b855b94-cb81-477d-9b17-0484597cc20e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:04.559975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2b855b94-cb81-477d-9b17-0484597cc20e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:04.656767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 720/1430] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:30:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:04.807847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec5c221-9144-4c0e-b3b9-ab5e482f0e25 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:04.809312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7ec5c221-9144-4c0e-b3b9-ab5e482f0e25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:04.858676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:04.956790 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec5c221-9144-4c0e-b3b9-ab5e482f0e25 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:05.068232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da52c6a1-9e38-4768-b16c-5c9c3da5112a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:05.069338 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da52c6a1-9e38-4768-b16c-5c9c3da5112a service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:05.069657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da52c6a1-9e38-4768-b16c-5c9c3da5112a service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:05.070019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da52c6a1-9e38-4768-b16c-5c9c3da5112a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:05.071598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-da52c6a1-9e38-4768-b16c-5c9c3da5112a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:05.078670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:05.159945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da52c6a1-9e38-4768-b16c-5c9c3da5112a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:05.161567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da52c6a1-9e38-4768-b16c-5c9c3da5112a service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:05.179081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 711/1431] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:06.237105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3f7b8d-8577-4efb-be36-4edfddeec695 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:06.237659 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3f7b8d-8577-4efb-be36-4edfddeec695 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:06.237659 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3f7b8d-8577-4efb-be36-4edfddeec695 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:06.238989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 721/1432] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:30:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:06.245998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7604ab7-ba8f-459d-827f-6cd9b523eace None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:06.246469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7604ab7-ba8f-459d-827f-6cd9b523eace None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:06.246576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7604ab7-ba8f-459d-827f-6cd9b523eace None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:06.324396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b7604ab7-ba8f-459d-827f-6cd9b523eace None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:06.325600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b7604ab7-ba8f-459d-827f-6cd9b523eace None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:06.393386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:06.422436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 712/1433] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:30:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:06.573459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3f7b8d-8577-4efb-be36-4edfddeec695 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:06.574908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8b3f7b8d-8577-4efb-be36-4edfddeec695 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:06.593277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:06.721825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3f7b8d-8577-4efb-be36-4edfddeec695 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:06.807801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:06.833522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335aaabb-bfaf-4c42-be85-badb400af0a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:06.834078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335aaabb-bfaf-4c42-be85-badb400af0a8 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:06.834469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335aaabb-bfaf-4c42-be85-badb400af0a8 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:06.834935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335aaabb-bfaf-4c42-be85-badb400af0a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:06.836554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-335aaabb-bfaf-4c42-be85-badb400af0a8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:06.924666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335aaabb-bfaf-4c42-be85-badb400af0a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:06.926255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335aaabb-bfaf-4c42-be85-badb400af0a8 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:06.944742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 722/1434] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:08.011143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3905169f-b613-44af-9f4d-67b3159a9781 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:08.011717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3905169f-b613-44af-9f4d-67b3159a9781 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:08.011717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3905169f-b613-44af-9f4d-67b3159a9781 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:08.013564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 713/1435] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:30:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:08.019982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4b9c20-8721-49f1-983f-4faa268cd331 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:08.020469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4b9c20-8721-49f1-983f-4faa268cd331 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:08.020580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4b9c20-8721-49f1-983f-4faa268cd331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:08.096252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-af4b9c20-8721-49f1-983f-4faa268cd331 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:08.097499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-af4b9c20-8721-49f1-983f-4faa268cd331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:08.139089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:08.194124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 723/1436] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:30:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:08.345552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3905169f-b613-44af-9f4d-67b3159a9781 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:08.347077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3905169f-b613-44af-9f4d-67b3159a9781 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:08.492267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3905169f-b613-44af-9f4d-67b3159a9781 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:08.546857 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:08.606163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d11a93-94bc-459b-af92-60a3a59b9c45 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:08.606763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d11a93-94bc-459b-af92-60a3a59b9c45 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:08.607043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d11a93-94bc-459b-af92-60a3a59b9c45 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:08.607315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d11a93-94bc-459b-af92-60a3a59b9c45 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:08.608968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-03d11a93-94bc-459b-af92-60a3a59b9c45 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:08.697483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d11a93-94bc-459b-af92-60a3a59b9c45 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:08.699085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d11a93-94bc-459b-af92-60a3a59b9c45 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:08.718259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 714/1437] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:09.776952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9008140-a5ca-46ac-a09f-8239b0abaad1 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:09.777522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9008140-a5ca-46ac-a09f-8239b0abaad1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:09.777522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9008140-a5ca-46ac-a09f-8239b0abaad1 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:09.778865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 724/1438] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:30:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:09.785356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1212840-4fa3-43cb-a959-0f3ec2418666 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:09.785787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1212840-4fa3-43cb-a959-0f3ec2418666 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:09.785913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1212840-4fa3-43cb-a959-0f3ec2418666 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:09.864938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b1212840-4fa3-43cb-a959-0f3ec2418666 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:09.866242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b1212840-4fa3-43cb-a959-0f3ec2418666 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:09.868230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:09.962702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 715/1439] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:30:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:10.112042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9008140-a5ca-46ac-a09f-8239b0abaad1 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:10.113580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a9008140-a5ca-46ac-a09f-8239b0abaad1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:10.259325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9008140-a5ca-46ac-a09f-8239b0abaad1 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:10.285618 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:10.372435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb53b6d-82ce-4eda-8dd9-b77f55a9e9d4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:10.372959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb53b6d-82ce-4eda-8dd9-b77f55a9e9d4 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:10.373254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb53b6d-82ce-4eda-8dd9-b77f55a9e9d4 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:10.373505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb53b6d-82ce-4eda-8dd9-b77f55a9e9d4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:10.375099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5eb53b6d-82ce-4eda-8dd9-b77f55a9e9d4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:10.465757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb53b6d-82ce-4eda-8dd9-b77f55a9e9d4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:10.467340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb53b6d-82ce-4eda-8dd9-b77f55a9e9d4 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:10.485083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 725/1440] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:12.550268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413c5a7a-8b68-4b0b-81e4-83f4b37d9c6b None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:12.550812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413c5a7a-8b68-4b0b-81e4-83f4b37d9c6b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:12.550812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413c5a7a-8b68-4b0b-81e4-83f4b37d9c6b None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:12.552220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 716/1441] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:30:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:12.558204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0900b34-81c3-4297-9c61-6d05e812217c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:12.558616 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0900b34-81c3-4297-9c61-6d05e812217c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:12.558749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0900b34-81c3-4297-9c61-6d05e812217c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:12.599050 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:12.625151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b0900b34-81c3-4297-9c61-6d05e812217c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:12.626370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b0900b34-81c3-4297-9c61-6d05e812217c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:12.721881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 726/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:30:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:13.010849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413c5a7a-8b68-4b0b-81e4-83f4b37d9c6b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:13.012401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-413c5a7a-8b68-4b0b-81e4-83f4b37d9c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:13.116640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413c5a7a-8b68-4b0b-81e4-83f4b37d9c6b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:13.149503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:13.190573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3291d5c-51c6-44b8-95b6-9742b376ac8e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:13.191169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3291d5c-51c6-44b8-95b6-9742b376ac8e service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:13.191449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3291d5c-51c6-44b8-95b6-9742b376ac8e service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:13.191723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3291d5c-51c6-44b8-95b6-9742b376ac8e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:13.193871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c3291d5c-51c6-44b8-95b6-9742b376ac8e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:13.302606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3291d5c-51c6-44b8-95b6-9742b376ac8e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:13.304034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3291d5c-51c6-44b8-95b6-9742b376ac8e service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:13.323389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 717/1443] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:15.638572 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890fce84-1f48-4aba-bee1-4c2d622422b3 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:15.639178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890fce84-1f48-4aba-bee1-4c2d622422b3 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:15.639178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890fce84-1f48-4aba-bee1-4c2d622422b3 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:15.640501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 727/1444] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:30:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:15.645794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8b86c4-731e-4009-9d25-4d58a2b40538 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:15.646200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8b86c4-731e-4009-9d25-4d58a2b40538 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:15.646421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8b86c4-731e-4009-9d25-4d58a2b40538 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:15.713070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3a8b86c4-731e-4009-9d25-4d58a2b40538 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:15.714287 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3a8b86c4-731e-4009-9d25-4d58a2b40538 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:15.727883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:15.812407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 718/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:30:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:16.096310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890fce84-1f48-4aba-bee1-4c2d622422b3 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:16.097881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-890fce84-1f48-4aba-bee1-4c2d622422b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:16.202732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890fce84-1f48-4aba-bee1-4c2d622422b3 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:16.215600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:16.277348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b868361-b05e-4abb-82f3-39a4a1d84e82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:16.277951 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b868361-b05e-4abb-82f3-39a4a1d84e82 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:16.278233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b868361-b05e-4abb-82f3-39a4a1d84e82 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:16.278506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b868361-b05e-4abb-82f3-39a4a1d84e82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:16.280095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8b868361-b05e-4abb-82f3-39a4a1d84e82 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:16.389840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b868361-b05e-4abb-82f3-39a4a1d84e82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:16.391370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b868361-b05e-4abb-82f3-39a4a1d84e82 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:16.409296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 728/1446] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:19.050098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309ea71-d4e2-4264-87a7-3d471a581006 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:19.050489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309ea71-d4e2-4264-87a7-3d471a581006 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:19.050646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309ea71-d4e2-4264-87a7-3d471a581006 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:19.052004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 719/1447] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:30:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:19.057981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378ccbeb-ab05-450d-bf32-d3f574df2da2 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:19.058424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378ccbeb-ab05-450d-bf32-d3f574df2da2 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:19.058530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378ccbeb-ab05-450d-bf32-d3f574df2da2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:19.106826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:19.124771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-378ccbeb-ab05-450d-bf32-d3f574df2da2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:19.125978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-378ccbeb-ab05-450d-bf32-d3f574df2da2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:19.222297 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 729/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:30:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:19.510241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309ea71-d4e2-4264-87a7-3d471a581006 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:19.511777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a309ea71-d4e2-4264-87a7-3d471a581006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:19.613484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309ea71-d4e2-4264-87a7-3d471a581006 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:19.653725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:19.688162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3cf6d-7123-4d03-ac46-173b1d324c97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:19.688869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3cf6d-7123-4d03-ac46-173b1d324c97 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:19.689146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3cf6d-7123-4d03-ac46-173b1d324c97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:19.689421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3cf6d-7123-4d03-ac46-173b1d324c97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:19.690994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1ba3cf6d-7123-4d03-ac46-173b1d324c97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:19.809136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba3cf6d-7123-4d03-ac46-173b1d324c97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:19.810556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba3cf6d-7123-4d03-ac46-173b1d324c97 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:19.830041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 720/1449] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:22.124409 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2862dc-8e9a-4d92-8f19-c7e4516ac0d9 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:22.124930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2862dc-8e9a-4d92-8f19-c7e4516ac0d9 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:22.125053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2862dc-8e9a-4d92-8f19-c7e4516ac0d9 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:22.126286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 730/1450] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:30:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:22.131796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de637a-d01a-491a-83b5-93f67e1bbb8b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:22.132224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de637a-d01a-491a-83b5-93f67e1bbb8b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:22.132402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de637a-d01a-491a-83b5-93f67e1bbb8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:22.199267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-80de637a-d01a-491a-83b5-93f67e1bbb8b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:22.200420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-80de637a-d01a-491a-83b5-93f67e1bbb8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:22.222281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:22.299744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 721/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:30:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:22.590168 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2862dc-8e9a-4d92-8f19-c7e4516ac0d9 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:22.591738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4f2862dc-8e9a-4d92-8f19-c7e4516ac0d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:22.693272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2862dc-8e9a-4d92-8f19-c7e4516ac0d9 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:22.714135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:22.766482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12a47c6-9922-490e-9047-bf2447ec8924 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:22.767083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a47c6-9922-490e-9047-bf2447ec8924 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:22.767362 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a47c6-9922-490e-9047-bf2447ec8924 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:22.767633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a47c6-9922-490e-9047-bf2447ec8924 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:22.769262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c12a47c6-9922-490e-9047-bf2447ec8924 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:22.879771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12a47c6-9922-490e-9047-bf2447ec8924 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:22.881777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12a47c6-9922-490e-9047-bf2447ec8924 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:22.899919 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 731/1452] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:25.118482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53030ccf-6b17-43eb-9e69-3c3abfa5c98c None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:25.119030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53030ccf-6b17-43eb-9e69-3c3abfa5c98c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:25.119030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53030ccf-6b17-43eb-9e69-3c3abfa5c98c None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:25.120443 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 722/1453] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:30:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:25.125727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1d85d9-db93-4d2b-b19c-b14d75895c94 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:25.126154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1d85d9-db93-4d2b-b19c-b14d75895c94 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:25.126278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1d85d9-db93-4d2b-b19c-b14d75895c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:25.188977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:25.192017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-bc1d85d9-db93-4d2b-b19c-b14d75895c94 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:25.193834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-bc1d85d9-db93-4d2b-b19c-b14d75895c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:25.290047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 732/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:30:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:25.572636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53030ccf-6b17-43eb-9e69-3c3abfa5c98c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:25.574233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-53030ccf-6b17-43eb-9e69-3c3abfa5c98c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:25.677913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53030ccf-6b17-43eb-9e69-3c3abfa5c98c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:25.738677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:25.763012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907abb30-d6f9-4a2a-8c98-80b4298eed49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:25.763614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907abb30-d6f9-4a2a-8c98-80b4298eed49 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:25.763893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907abb30-d6f9-4a2a-8c98-80b4298eed49 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:25.764164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907abb30-d6f9-4a2a-8c98-80b4298eed49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:25.765783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-907abb30-d6f9-4a2a-8c98-80b4298eed49 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:25.875621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907abb30-d6f9-4a2a-8c98-80b4298eed49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:25.877027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907abb30-d6f9-4a2a-8c98-80b4298eed49 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:25.895336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 723/1455] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:28.145844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ddc45b-b84e-456a-b520-40043939ad45 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:28.146406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ddc45b-b84e-456a-b520-40043939ad45 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:28.146532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ddc45b-b84e-456a-b520-40043939ad45 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:28.147878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 733/1456] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:30:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:28.153429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf1b3be-6910-4317-9945-f2d3988f9dea None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:28.153844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf1b3be-6910-4317-9945-f2d3988f9dea None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:28.154018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf1b3be-6910-4317-9945-f2d3988f9dea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:28.221252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-adf1b3be-6910-4317-9945-f2d3988f9dea None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:28.222450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-adf1b3be-6910-4317-9945-f2d3988f9dea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:28.252152 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:28.319900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 724/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:30:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:28.612139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ddc45b-b84e-456a-b520-40043939ad45 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:28.613704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d5ddc45b-b84e-456a-b520-40043939ad45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:28.717828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ddc45b-b84e-456a-b520-40043939ad45 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:28.746794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:28.792173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fc33de-4207-44f4-b06c-0cc3500b9393 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:28.792804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc33de-4207-44f4-b06c-0cc3500b9393 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:28.793081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc33de-4207-44f4-b06c-0cc3500b9393 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:28.793350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc33de-4207-44f4-b06c-0cc3500b9393 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:28.794938 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-21fc33de-4207-44f4-b06c-0cc3500b9393 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:28.904056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fc33de-4207-44f4-b06c-0cc3500b9393 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:28.905436 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fc33de-4207-44f4-b06c-0cc3500b9393 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:28.923326 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 734/1458] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:31.117704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b93fc6-692e-427e-a719-8364e3e5738f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:31.118227 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b93fc6-692e-427e-a719-8364e3e5738f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:31.118345 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b93fc6-692e-427e-a719-8364e3e5738f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:31.119622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 725/1459] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:30:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:31.125095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b8ae9-41e7-4ffc-bf6b-e20244329ee3 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:31.125498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b8ae9-41e7-4ffc-bf6b-e20244329ee3 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:31.125768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b8ae9-41e7-4ffc-bf6b-e20244329ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:31.191641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-a23b8ae9-41e7-4ffc-bf6b-e20244329ee3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:31.192924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a23b8ae9-41e7-4ffc-bf6b-e20244329ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:31.201316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:31.291002 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 735/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:30:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:31.574779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b93fc6-692e-427e-a719-8364e3e5738f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:31.576318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-97b93fc6-692e-427e-a719-8364e3e5738f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:31.682891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b93fc6-692e-427e-a719-8364e3e5738f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:31.740408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:31.758552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68430e81-4a09-4291-9802-4dcf0461e822 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:31.759147 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68430e81-4a09-4291-9802-4dcf0461e822 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:31.759426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68430e81-4a09-4291-9802-4dcf0461e822 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:31.759695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68430e81-4a09-4291-9802-4dcf0461e822 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:31.761328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-68430e81-4a09-4291-9802-4dcf0461e822 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:31.870979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68430e81-4a09-4291-9802-4dcf0461e822 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:31.872352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68430e81-4a09-4291-9802-4dcf0461e822 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:31.890803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 726/1461] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:34.118431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d83ab-f8c4-48df-912d-d9fbee10d684 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:34.118936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d83ab-f8c4-48df-912d-d9fbee10d684 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:34.119031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d83ab-f8c4-48df-912d-d9fbee10d684 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:34.120480 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 736/1462] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:30:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:34.125894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f38279-dcc5-4996-83a6-7b91d984fdeb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:34.126317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f38279-dcc5-4996-83a6-7b91d984fdeb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:34.126446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f38279-dcc5-4996-83a6-7b91d984fdeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:34.194499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-21f38279-dcc5-4996-83a6-7b91d984fdeb None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:34.195628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-21f38279-dcc5-4996-83a6-7b91d984fdeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:34.236060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:34.294546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 727/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:30:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:34.583459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372d83ab-f8c4-48df-912d-d9fbee10d684 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:34.585131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-372d83ab-f8c4-48df-912d-d9fbee10d684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:34.687669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372d83ab-f8c4-48df-912d-d9fbee10d684 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:34.727202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:34.763523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e5ff8a-0409-447c-9d88-539dd1afac5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:34.764128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5ff8a-0409-447c-9d88-539dd1afac5b service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:34.764451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5ff8a-0409-447c-9d88-539dd1afac5b service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:34.764727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5ff8a-0409-447c-9d88-539dd1afac5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:34.766321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-34e5ff8a-0409-447c-9d88-539dd1afac5b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:34.875440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e5ff8a-0409-447c-9d88-539dd1afac5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:34.876842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e5ff8a-0409-447c-9d88-539dd1afac5b service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:34.894944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 737/1464] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:37.107054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f678f4-69f1-45ee-9d5f-0eb421fe2b23 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:37.107982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f678f4-69f1-45ee-9d5f-0eb421fe2b23 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:37.108293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f678f4-69f1-45ee-9d5f-0eb421fe2b23 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:37.109710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 728/1465] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:30:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:37.115521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2636b12-9e84-4c8b-8764-6bd89f615f9c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:37.115987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2636b12-9e84-4c8b-8764-6bd89f615f9c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:37.116093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2636b12-9e84-4c8b-8764-6bd89f615f9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:37.191862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b2636b12-9e84-4c8b-8764-6bd89f615f9c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:37.193365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b2636b12-9e84-4c8b-8764-6bd89f615f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:37.212473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:37.296589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 738/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:30:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:37.583004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f678f4-69f1-45ee-9d5f-0eb421fe2b23 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:37.584727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-08f678f4-69f1-45ee-9d5f-0eb421fe2b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:37.693438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f678f4-69f1-45ee-9d5f-0eb421fe2b23 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:37.768533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:37.776611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ffd3cf-cb9c-4e08-b429-6b0e3d84abc3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:37.777228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ffd3cf-cb9c-4e08-b429-6b0e3d84abc3 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:37.777558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ffd3cf-cb9c-4e08-b429-6b0e3d84abc3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:37.777803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ffd3cf-cb9c-4e08-b429-6b0e3d84abc3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:37.780099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b1ffd3cf-cb9c-4e08-b429-6b0e3d84abc3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:37.897430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ffd3cf-cb9c-4e08-b429-6b0e3d84abc3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:37.898944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ffd3cf-cb9c-4e08-b429-6b0e3d84abc3 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:37.918729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 729/1467] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:40.123032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edba37d5-7038-4abf-82b2-e9fcadaca5e4 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:40.123544 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edba37d5-7038-4abf-82b2-e9fcadaca5e4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:40.123544 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edba37d5-7038-4abf-82b2-e9fcadaca5e4 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:40.124863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 739/1468] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:30:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:30:40.130969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4c9636-87d9-41c0-b3d4-cac1cddc9171 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:40.131434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4c9636-87d9-41c0-b3d4-cac1cddc9171 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:40.131544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4c9636-87d9-41c0-b3d4-cac1cddc9171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:40.204457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4c4c9636-87d9-41c0-b3d4-cac1cddc9171 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:30:40.205660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4c4c9636-87d9-41c0-b3d4-cac1cddc9171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:40.257553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:40.312616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 730/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:30:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:30:40.600038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edba37d5-7038-4abf-82b2-e9fcadaca5e4 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:30:40.601705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-edba37d5-7038-4abf-82b2-e9fcadaca5e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:40.709471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edba37d5-7038-4abf-82b2-e9fcadaca5e4 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:30:40.759777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:40.788540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40e90e9-5015-4ce4-9ca8-186e0b684e0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:30:40.789147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40e90e9-5015-4ce4-9ca8-186e0b684e0f service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:30:40.789513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40e90e9-5015-4ce4-9ca8-186e0b684e0f service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:30:40.789788 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40e90e9-5015-4ce4-9ca8-186e0b684e0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:30:40.791413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b40e90e9-5015-4ce4-9ca8-186e0b684e0f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:30:40.907839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40e90e9-5015-4ce4-9ca8-186e0b684e0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:30:40.909241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40e90e9-5015-4ce4-9ca8-186e0b684e0f service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:30:40.928166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 740/1470] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:30:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:30:50.309632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=22 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:30:50.925523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=66 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:00.994881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2588ca59-2aa8-41eb-b7c0-f965f5a2378e None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:00.995439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2588ca59-2aa8-41eb-b7c0-f965f5a2378e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:00.995439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2588ca59-2aa8-41eb-b7c0-f965f5a2378e None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:00.996830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 731/1471] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:31:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:31:01.001952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c5278d-ed76-4e56-92da-c3dd7c59d986 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:01.002352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c5278d-ed76-4e56-92da-c3dd7c59d986 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:01.002498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c5278d-ed76-4e56-92da-c3dd7c59d986 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:01.074700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-70c5278d-ed76-4e56-92da-c3dd7c59d986 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:31:01.075900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-70c5278d-ed76-4e56-92da-c3dd7c59d986 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:01.172127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 741/1472] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:31:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:31:01.474543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b56470-ec6e-4a89-848c-a6ecac389bf6 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:01.474822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b56470-ec6e-4a89-848c-a6ecac389bf6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:01.475091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b56470-ec6e-4a89-848c-a6ecac389bf6 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:01.476433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 732/1473] 10.222.0.25 () {54 vars in 962 bytes} [Mon Sep 26 01:31:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:31:01.481377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19f7a6-13c8-41e8-bd61-16eaa12836d1 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:01.481655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19f7a6-13c8-41e8-bd61-16eaa12836d1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:01.481922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19f7a6-13c8-41e8-bd61-16eaa12836d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:01.557870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-5d19f7a6-13c8-41e8-bd61-16eaa12836d1 None None] MFA Rules not processed for user `e07b6890d3004dd78faed48a1ab90086`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:31:01.559076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5d19f7a6-13c8-41e8-bd61-16eaa12836d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:01.580221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:01.635321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 742/1474] 10.222.0.25 () {58 vars in 1059 bytes} [Mon Sep 26 01:31:01 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:31:01.643073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b56470-ec6e-4a89-848c-a6ecac389bf6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:31:01.644562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d2b56470-ec6e-4a89-848c-a6ecac389bf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:01.864705 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b56470-ec6e-4a89-848c-a6ecac389bf6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:31:01.936777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df42d435-00d3-48d4-8da9-2d80cf24b235 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e07b6890d3004dd78faed48a1ab90086', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:31:01.937378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df42d435-00d3-48d4-8da9-2d80cf24b235 service glance] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:01.937658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df42d435-00d3-48d4-8da9-2d80cf24b235 service glance] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:01.937933 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df42d435-00d3-48d4-8da9-2d80cf24b235 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:01.939522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-df42d435-00d3-48d4-8da9-2d80cf24b235 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:01.997980 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:02.057786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df42d435-00d3-48d4-8da9-2d80cf24b235 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:31:02.059229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df42d435-00d3-48d4-8da9-2d80cf24b235 service glance] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:31:02.077751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 733/1475] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:31:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:31:04.330316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9755d416-e015-4af1-b360-66f8c2804a76 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:04.330896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9755d416-e015-4af1-b360-66f8c2804a76 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:04.330896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9755d416-e015-4af1-b360-66f8c2804a76 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:04.332203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 743/1476] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:31:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:31:04.337989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c105d28-e033-49dc-8098-f4d92fdce841 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:04.338447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c105d28-e033-49dc-8098-f4d92fdce841 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:04.338544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c105d28-e033-49dc-8098-f4d92fdce841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:04.404002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2c105d28-e033-49dc-8098-f4d92fdce841 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:31:04.405239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2c105d28-e033-49dc-8098-f4d92fdce841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:04.501718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 734/1477] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:31:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:31:04.778018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9755d416-e015-4af1-b360-66f8c2804a76 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:31:04.779563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9755d416-e015-4af1-b360-66f8c2804a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:04.906907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9755d416-e015-4af1-b360-66f8c2804a76 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:31:04.956240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:05.000596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bfbff5-b0e1-42f4-85e0-15caad0f0e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:31:05.001199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bfbff5-b0e1-42f4-85e0-15caad0f0e34 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:05.001479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bfbff5-b0e1-42f4-85e0-15caad0f0e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:05.001746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bfbff5-b0e1-42f4-85e0-15caad0f0e34 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:05.002374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bfbff5-b0e1-42f4-85e0-15caad0f0e34 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:31:05.004747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: WARNING py.warnings [None req-f0bfbff5-b0e1-42f4-85e0-15caad0f0e34 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:31:05.004747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: warnings.warn(msg) Sep 26 01:31:05.004747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: Sep 26 01:31:05.005790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bfbff5-b0e1-42f4-85e0-15caad0f0e34 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:31:05.026132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 744/1478] 10.222.0.25 () {56 vars in 1108 bytes} [Mon Sep 26 01:31:04 2022] GET /identity/v3/projects => generated 2553 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:31:07.193017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd78979-ea8a-4412-b77a-68b16f82fcab None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:07.193607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd78979-ea8a-4412-b77a-68b16f82fcab None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:07.193607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd78979-ea8a-4412-b77a-68b16f82fcab None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:07.195000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 735/1479] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:31:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:31:07.200187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9bb8b2-eaeb-4c5a-b33a-bbf503037548 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:07.200690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9bb8b2-eaeb-4c5a-b33a-bbf503037548 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:07.200800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9bb8b2-eaeb-4c5a-b33a-bbf503037548 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:07.266817 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0f9bb8b2-eaeb-4c5a-b33a-bbf503037548 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:31:07.268021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0f9bb8b2-eaeb-4c5a-b33a-bbf503037548 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:07.364290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 745/1480] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:31:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:31:07.657180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd78979-ea8a-4412-b77a-68b16f82fcab None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:31:07.658723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1dd78979-ea8a-4412-b77a-68b16f82fcab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:07.668921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:07.806003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd78979-ea8a-4412-b77a-68b16f82fcab None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:31:07.885843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:07.918188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9c9d0a-e5be-495a-a612-3892ceb0e6de service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:31:07.918714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c9d0a-e5be-495a-a612-3892ceb0e6de service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:07.918992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c9d0a-e5be-495a-a612-3892ceb0e6de service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:07.919262 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9c9d0a-e5be-495a-a612-3892ceb0e6de service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:07.920899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-af9c9d0a-e5be-495a-a612-3892ceb0e6de service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:08.010241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9c9d0a-e5be-495a-a612-3892ceb0e6de service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:31:08.011820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9c9d0a-e5be-495a-a612-3892ceb0e6de service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:31:08.029455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 736/1481] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:31:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:31:10.387336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9221be-1be5-499c-b4f1-38990d028745 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:10.387898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9221be-1be5-499c-b4f1-38990d028745 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:10.387898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9221be-1be5-499c-b4f1-38990d028745 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:10.389248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 746/1482] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:31:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:31:10.394469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf514ea2-f321-49ca-bde3-b310e420f4cc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:10.394941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf514ea2-f321-49ca-bde3-b310e420f4cc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:10.395185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf514ea2-f321-49ca-bde3-b310e420f4cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:10.461139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bf514ea2-f321-49ca-bde3-b310e420f4cc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:31:10.462353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bf514ea2-f321-49ca-bde3-b310e420f4cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:10.508608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:10.558409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 737/1483] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:31:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:31:10.846120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9221be-1be5-499c-b4f1-38990d028745 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:31:10.848221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-da9221be-1be5-499c-b4f1-38990d028745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:10.872110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:10.951302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9221be-1be5-499c-b4f1-38990d028745 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:31:11.024607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09917c73-7d01-46c0-8e75-8a808b291592 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:31:11.025205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09917c73-7d01-46c0-8e75-8a808b291592 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:11.025482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09917c73-7d01-46c0-8e75-8a808b291592 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:11.025752 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09917c73-7d01-46c0-8e75-8a808b291592 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:11.027341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-09917c73-7d01-46c0-8e75-8a808b291592 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:11.103059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:11.138001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09917c73-7d01-46c0-8e75-8a808b291592 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:31:11.139543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09917c73-7d01-46c0-8e75-8a808b291592 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:31:11.158393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 747/1484] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:31:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:31:13.380459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0843c8-bd4c-4ced-82cb-1879d85276d2 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:13.381001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0843c8-bd4c-4ced-82cb-1879d85276d2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:13.381001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0843c8-bd4c-4ced-82cb-1879d85276d2 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:13.382239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 738/1485] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:31:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:31:13.387678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7551171a-9371-4777-8d8f-4cf9c2b97e01 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:13.388127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7551171a-9371-4777-8d8f-4cf9c2b97e01 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:13.388248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7551171a-9371-4777-8d8f-4cf9c2b97e01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:13.453301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7551171a-9371-4777-8d8f-4cf9c2b97e01 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:31:13.454504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7551171a-9371-4777-8d8f-4cf9c2b97e01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:13.549182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 748/1486] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:31:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:31:13.842549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0843c8-bd4c-4ced-82cb-1879d85276d2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:31:13.844076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ca0843c8-bd4c-4ced-82cb-1879d85276d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:13.945913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0843c8-bd4c-4ced-82cb-1879d85276d2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:31:14.017505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:14.018962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3597c403-7602-46be-8cd3-8f1f8b47b1eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:31:14.020151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3597c403-7602-46be-8cd3-8f1f8b47b1eb service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:14.020537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3597c403-7602-46be-8cd3-8f1f8b47b1eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:14.021004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3597c403-7602-46be-8cd3-8f1f8b47b1eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:14.023002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3597c403-7602-46be-8cd3-8f1f8b47b1eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:14.131171 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3597c403-7602-46be-8cd3-8f1f8b47b1eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:31:14.132565 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3597c403-7602-46be-8cd3-8f1f8b47b1eb service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:31:14.154689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 739/1487] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:31:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:31:16.443835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f8dae-0d5e-4a46-8974-0e1477ff1414 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:16.444426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f8dae-0d5e-4a46-8974-0e1477ff1414 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:16.444426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f8dae-0d5e-4a46-8974-0e1477ff1414 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:16.445750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 749/1488] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:31:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:31:16.450970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49f6c1a-cb52-4aff-8f1d-3869533008a5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:16.451403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49f6c1a-cb52-4aff-8f1d-3869533008a5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:16.451516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49f6c1a-cb52-4aff-8f1d-3869533008a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:16.516991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b49f6c1a-cb52-4aff-8f1d-3869533008a5 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:31:16.518198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b49f6c1a-cb52-4aff-8f1d-3869533008a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:16.570536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:16.615820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 740/1489] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:31:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:31:16.907499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982f8dae-0d5e-4a46-8974-0e1477ff1414 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:31:16.909136 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-982f8dae-0d5e-4a46-8974-0e1477ff1414 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:16.946405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:17.014464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982f8dae-0d5e-4a46-8974-0e1477ff1414 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:31:17.088061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1357e3b8-1465-4529-9b16-651abb63719d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:31:17.088688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1357e3b8-1465-4529-9b16-651abb63719d service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:17.088963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1357e3b8-1465-4529-9b16-651abb63719d service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:17.089227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1357e3b8-1465-4529-9b16-651abb63719d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:17.090812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1357e3b8-1465-4529-9b16-651abb63719d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:17.172074 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:17.201040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1357e3b8-1465-4529-9b16-651abb63719d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:31:17.202732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1357e3b8-1465-4529-9b16-651abb63719d service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:31:17.221100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 750/1490] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:31:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:31:19.698713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386ff91d-d1b3-4e7b-92c4-5e4263b3e347 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:19.699262 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386ff91d-d1b3-4e7b-92c4-5e4263b3e347 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:19.699262 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386ff91d-d1b3-4e7b-92c4-5e4263b3e347 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:19.700560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 741/1491] 10.222.0.25 () {54 vars in 927 bytes} [Mon Sep 26 01:31:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:31:19.705968 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58805c05-2f00-4b1c-9018-a31b99e3128b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:19.706362 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58805c05-2f00-4b1c-9018-a31b99e3128b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:19.706509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58805c05-2f00-4b1c-9018-a31b99e3128b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:19.787686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-58805c05-2f00-4b1c-9018-a31b99e3128b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:31:19.789810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-58805c05-2f00-4b1c-9018-a31b99e3128b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:19.888662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 751/1492] 10.222.0.25 () {58 vars in 1024 bytes} [Mon Sep 26 01:31:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:31:20.176317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386ff91d-d1b3-4e7b-92c4-5e4263b3e347 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:31:20.177881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-386ff91d-d1b3-4e7b-92c4-5e4263b3e347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:20.280002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386ff91d-d1b3-4e7b-92c4-5e4263b3e347 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:31:20.354759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516e0766-f8b8-4d6c-94e7-66a81a37d056 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:31:20.355271 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e0766-f8b8-4d6c-94e7-66a81a37d056 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:31:20.355546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e0766-f8b8-4d6c-94e7-66a81a37d056 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:31:20.355813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516e0766-f8b8-4d6c-94e7-66a81a37d056 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:31:20.357433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-516e0766-f8b8-4d6c-94e7-66a81a37d056 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:31:20.367078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:20.466348 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516e0766-f8b8-4d6c-94e7-66a81a37d056 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:31:20.467769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516e0766-f8b8-4d6c-94e7-66a81a37d056 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:31:20.486045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 742/1493] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:31:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:31:29.887131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=82 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:31:30.483748 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=50 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:34.652039 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b11b25-dd0a-4377-8a91-b6e05bcf0e75 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:34.652685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b11b25-dd0a-4377-8a91-b6e05bcf0e75 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:34.652685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b11b25-dd0a-4377-8a91-b6e05bcf0e75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:34.728080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-36b11b25-dd0a-4377-8a91-b6e05bcf0e75 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:32:34.729443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-36b11b25-dd0a-4377-8a91-b6e05bcf0e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:34.826518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 752/1494] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:32:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:32:34.835107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef566be-73bc-485c-8fcc-8f2dcfa3a3c7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:34.835588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef566be-73bc-485c-8fcc-8f2dcfa3a3c7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:34.835685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef566be-73bc-485c-8fcc-8f2dcfa3a3c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:34.911760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5ef566be-73bc-485c-8fcc-8f2dcfa3a3c7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:32:34.912932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5ef566be-73bc-485c-8fcc-8f2dcfa3a3c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:35.009123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 743/1495] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:32:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:32:35.013746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b11b25-dd0a-4377-8a91-b6e05bcf0e75 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:32:35.014952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-36b11b25-dd0a-4377-8a91-b6e05bcf0e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:35.027009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b11b25-dd0a-4377-8a91-b6e05bcf0e75 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:32:35.088200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:35.122402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:32:35.122996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:35.123269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:35.123535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:35.124171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:32:35.124494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:32:35.125986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: WARNING py.warnings [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:32:35.125986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: warnings.warn(msg) Sep 26 01:32:35.125986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: Sep 26 01:32:35.127256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:32:35.138159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 753/1496] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:32:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:32:35.147119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef566be-73bc-485c-8fcc-8f2dcfa3a3c7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:32:35.156169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef566be-73bc-485c-8fcc-8f2dcfa3a3c7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:32:35.218943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:35.253610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f688ec3b-4850-4320-82e7-c6858360e155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:32:35.254211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f688ec3b-4850-4320-82e7-c6858360e155 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:35.254483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f688ec3b-4850-4320-82e7-c6858360e155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:35.254748 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f688ec3b-4850-4320-82e7-c6858360e155 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:35.255377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f688ec3b-4850-4320-82e7-c6858360e155 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:32:35.257145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-f688ec3b-4850-4320-82e7-c6858360e155 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:32:35.257145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 26 01:32:35.257145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: Sep 26 01:32:35.258339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f688ec3b-4850-4320-82e7-c6858360e155 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:32:35.315167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 744/1497] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:32:35 2022] POST /identity/v3/projects => generated 377 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:32:35.319398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:32:35.320934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:35.354936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e385e1-79d0-4b03-9097-33d48c2609c3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:32:35.448220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6cac7c-05ce-4416-bd28-71f3475fa75c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:32:35.448836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6cac7c-05ce-4416-bd28-71f3475fa75c admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:35.449112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6cac7c-05ce-4416-bd28-71f3475fa75c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:35.449376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6cac7c-05ce-4416-bd28-71f3475fa75c admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:35.449987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6cac7c-05ce-4416-bd28-71f3475fa75c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:32:35.450982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: WARNING py.warnings [None req-9d6cac7c-05ce-4416-bd28-71f3475fa75c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:32:35.450982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: warnings.warn(msg) Sep 26 01:32:35.450982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: Sep 26 01:32:35.452472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6cac7c-05ce-4416-bd28-71f3475fa75c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:32:35.494990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 754/1498] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:32:35 2022] POST /identity/v3/users => generated 289 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:32:35.499203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f688ec3b-4850-4320-82e7-c6858360e155 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:32:35.500814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f688ec3b-4850-4320-82e7-c6858360e155 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:35.564926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f688ec3b-4850-4320-82e7-c6858360e155 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:32:35.650021 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:35.661369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587636d4-2914-473b-968b-87e2a7e5e851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:32:35.661969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587636d4-2914-473b-968b-87e2a7e5e851 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:35.662248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587636d4-2914-473b-968b-87e2a7e5e851 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:35.662522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587636d4-2914-473b-968b-87e2a7e5e851 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:35.663166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587636d4-2914-473b-968b-87e2a7e5e851 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:32:35.664181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-587636d4-2914-473b-968b-87e2a7e5e851 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:32:35.664181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 26 01:32:35.664181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: Sep 26 01:32:35.665095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587636d4-2914-473b-968b-87e2a7e5e851 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:32:35.677486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 745/1499] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:32:35 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:32:35.681849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6cac7c-05ce-4416-bd28-71f3475fa75c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:32:35.683404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9d6cac7c-05ce-4416-bd28-71f3475fa75c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:35.695427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6cac7c-05ce-4416-bd28-71f3475fa75c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:32:35.721117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:35.791780 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae82895-b7b7-40e3-909a-79cb6278251e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:32:35.792327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae82895-b7b7-40e3-909a-79cb6278251e admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:35.792633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae82895-b7b7-40e3-909a-79cb6278251e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:35.792898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae82895-b7b7-40e3-909a-79cb6278251e admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:35.793531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae82895-b7b7-40e3-909a-79cb6278251e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:32:35.794552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: WARNING py.warnings [None req-2ae82895-b7b7-40e3-909a-79cb6278251e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:32:35.794552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: warnings.warn(msg) Sep 26 01:32:35.794552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: Sep 26 01:32:35.795429 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae82895-b7b7-40e3-909a-79cb6278251e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:32:35.807658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 755/1500] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:32:35 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:32:35.811969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587636d4-2914-473b-968b-87e2a7e5e851 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:32:35.814200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-587636d4-2914-473b-968b-87e2a7e5e851 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:35.848142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587636d4-2914-473b-968b-87e2a7e5e851 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:32:35.941127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9e052f-6249-4df4-9d3f-a991146803a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:32:35.941774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9e052f-6249-4df4-9d3f-a991146803a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:35.942049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9e052f-6249-4df4-9d3f-a991146803a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:35.942319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9e052f-6249-4df4-9d3f-a991146803a7 admin admin] PATH_INFO: `/v3/projects/fb230986fb7544cd8349f02096f7af81/users/433e0f13d844414cab626171b99fa206/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:35.965365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9e052f-6249-4df4-9d3f-a991146803a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb230986fb7544cd8349f02096f7af81, user_id=433e0f13d844414cab626171b99fa206, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:32:35.966387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-0b9e052f-6249-4df4-9d3f-a991146803a7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:32:35.966387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 26 01:32:35.966387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: Sep 26 01:32:35.968326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9e052f-6249-4df4-9d3f-a991146803a7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:32:35.993168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 746/1501] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:32:35 2022] PUT /identity/v3/projects/fb230986fb7544cd8349f02096f7af81/users/433e0f13d844414cab626171b99fa206/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:32:36.006035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c10b7f7-a23e-41cf-b06a-ebed5b41b282 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:36.006341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c10b7f7-a23e-41cf-b06a-ebed5b41b282 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:36.006619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c10b7f7-a23e-41cf-b06a-ebed5b41b282 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:36.064484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0c10b7f7-a23e-41cf-b06a-ebed5b41b282 None None] MFA Rules not processed for user `433e0f13d844414cab626171b99fa206`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:32:36.065700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0c10b7f7-a23e-41cf-b06a-ebed5b41b282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:36.142604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 756/1502] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:32:36 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:32:36.156295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0fe5b0-1d67-47a9-ad95-b1c59499a034 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:36.156622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0fe5b0-1d67-47a9-ad95-b1c59499a034 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:36.156939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0fe5b0-1d67-47a9-ad95-b1c59499a034 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:36.158306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 747/1503] 10.222.0.25 () {54 vars in 963 bytes} [Mon Sep 26 01:32:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:32:36.163239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ed83f-17b7-4b53-93fb-eb8b5b56d36b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:36.163521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ed83f-17b7-4b53-93fb-eb8b5b56d36b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:36.163794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ed83f-17b7-4b53-93fb-eb8b5b56d36b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:36.196569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:36.239984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b70ed83f-17b7-4b53-93fb-eb8b5b56d36b None None] MFA Rules not processed for user `2bb5f76eb08741f3a8c7a3a140e01e24`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:32:36.241221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b70ed83f-17b7-4b53-93fb-eb8b5b56d36b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:36.316375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 757/1504] 10.222.0.25 () {58 vars in 1060 bytes} [Mon Sep 26 01:32:36 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:32:36.326046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0fe5b0-1d67-47a9-ad95-b1c59499a034 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:32:36.327836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ff0fe5b0-1d67-47a9-ad95-b1c59499a034 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:36.379588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:36.434011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0fe5b0-1d67-47a9-ad95-b1c59499a034 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:32:36.507561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29b93c-d01e-4171-82ef-7db5d1060565 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:32:36.508153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29b93c-d01e-4171-82ef-7db5d1060565 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:36.508461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29b93c-d01e-4171-82ef-7db5d1060565 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:36.508732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29b93c-d01e-4171-82ef-7db5d1060565 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:36.510316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1f29b93c-d01e-4171-82ef-7db5d1060565 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:36.607233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29b93c-d01e-4171-82ef-7db5d1060565 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:32:36.608620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29b93c-d01e-4171-82ef-7db5d1060565 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:32:36.612055 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:36.627470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 748/1505] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:32:36 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:32:36.673953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c2bccf-93ae-4d77-95af-27be1aa9e164 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:36.674245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c2bccf-93ae-4d77-95af-27be1aa9e164 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:36.674524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c2bccf-93ae-4d77-95af-27be1aa9e164 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:36.675770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 758/1506] 10.222.0.25 () {52 vars in 830 bytes} [Mon Sep 26 01:32:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:32:36.679342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29b93c-d01e-4171-82ef-7db5d1060565 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:32:36.680582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1f29b93c-d01e-4171-82ef-7db5d1060565 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:36.779146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29b93c-d01e-4171-82ef-7db5d1060565 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:32:36.871238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:32:36.871862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:36.872042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:36.872313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] PATH_INFO: `/v3/users/433e0f13d844414cab626171b99fa206` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:36.885514 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] RBAC: Authorizing `identity:delete_user(user_id=433e0f13d844414cab626171b99fa206)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:32:36.886527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:32:36.886527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 26 01:32:36.886527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: Sep 26 01:32:36.887729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:32:36.897308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:36.953589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '433e0f13d844414cab626171b99fa206'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:32:36.962943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '433e0f13d844414cab626171b99fa206'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:32:36.963585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '433e0f13d844414cab626171b99fa206'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:32:36.964001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '433e0f13d844414cab626171b99fa206'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:32:36.978523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '433e0f13d844414cab626171b99fa206'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:32:37.008136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 749/1507] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:32:36 2022] DELETE /identity/v3/users/433e0f13d844414cab626171b99fa206 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:32:37.012486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f88310-0527-4915-927c-7d4f2573916d None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:37.012950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f88310-0527-4915-927c-7d4f2573916d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:37.013045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f88310-0527-4915-927c-7d4f2573916d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:37.071666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-79f88310-0527-4915-927c-7d4f2573916d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:32:37.072905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-79f88310-0527-4915-927c-7d4f2573916d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:37.146764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:37.160213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 759/1508] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:32:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:32:37.169885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:32:37.171094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:37.270106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89e09e0-4720-4239-ab55-c1f0ed55e500 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:32:37.325164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:37.346505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e28c978-a83a-4a31-a738-135c356b51e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:32:37.347120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e28c978-a83a-4a31-a738-135c356b51e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:37.347427 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e28c978-a83a-4a31-a738-135c356b51e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:37.347700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e28c978-a83a-4a31-a738-135c356b51e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:37.349348 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6e28c978-a83a-4a31-a738-135c356b51e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:37.458518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e28c978-a83a-4a31-a738-135c356b51e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:32:37.459963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e28c978-a83a-4a31-a738-135c356b51e3 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:32:37.480003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 750/1509] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:32:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:32:37.833101 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f88310-0527-4915-927c-7d4f2573916d None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:32:37.834332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-79f88310-0527-4915-927c-7d4f2573916d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:32:37.846152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f88310-0527-4915-927c-7d4f2573916d None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:32:37.941879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:32:37.942505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:32:37.942781 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:32:37.943049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] PATH_INFO: `/v3/projects/fb230986fb7544cd8349f02096f7af81` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:32:37.953220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb230986fb7544cd8349f02096f7af81)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:32:37.955068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: WARNING py.warnings [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:32:37.955068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: warnings.warn(msg) Sep 26 01:32:37.955068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: Sep 26 01:32:37.956306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:32:38.015024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb230986fb7544cd8349f02096f7af81', 'request_id': 'req-08e570f8-ee38-4790-9f26-6df682692539'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:32:38.028791 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb230986fb7544cd8349f02096f7af81', 'request_id': 'req-08e570f8-ee38-4790-9f26-6df682692539'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:32:38.034823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb230986fb7544cd8349f02096f7af81', 'request_id': 'req-08e570f8-ee38-4790-9f26-6df682692539'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:32:38.060577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] The token cache is being invalidate because project fb230986fb7544cd8349f02096f7af81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:32:38.060873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:32:38.061393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08e570f8-ee38-4790-9f26-6df682692539 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:32:38.062691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 760/1510] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:32:37 2022] DELETE /identity/v3/projects/fb230986fb7544cd8349f02096f7af81 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:32:47.477800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=62 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:32:48.058446 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:34:06.775890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188d33ba-5dcc-4713-9b09-2665ac44f0e0 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:06.776547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188d33ba-5dcc-4713-9b09-2665ac44f0e0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:06.776547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188d33ba-5dcc-4713-9b09-2665ac44f0e0 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:06.778432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 751/1511] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:34:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:34:06.783280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad8fc1b-d32a-4e02-afaa-f91e3c141379 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:06.783693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad8fc1b-d32a-4e02-afaa-f91e3c141379 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:06.783812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad8fc1b-d32a-4e02-afaa-f91e3c141379 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:06.863672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7ad8fc1b-d32a-4e02-afaa-f91e3c141379 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:34:06.864914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7ad8fc1b-d32a-4e02-afaa-f91e3c141379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:06.966815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 761/1512] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:34:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:34:07.121208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188d33ba-5dcc-4713-9b09-2665ac44f0e0 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:34:07.123960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-188d33ba-5dcc-4713-9b09-2665ac44f0e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:07.245949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188d33ba-5dcc-4713-9b09-2665ac44f0e0 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:34:07.251519 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925e2706-f247-467d-bad9-26661df22adf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:34:07.252127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925e2706-f247-467d-bad9-26661df22adf service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:07.252441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925e2706-f247-467d-bad9-26661df22adf service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:07.252718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925e2706-f247-467d-bad9-26661df22adf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:07.260865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925e2706-f247-467d-bad9-26661df22adf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:34:07.262437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925e2706-f247-467d-bad9-26661df22adf service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:34:07.271709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 752/1513] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:34:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:34:08.361426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2675d15c-ec2d-439e-88f2-f314247a2fe4 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:08.361965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2675d15c-ec2d-439e-88f2-f314247a2fe4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:08.361965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2675d15c-ec2d-439e-88f2-f314247a2fe4 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:08.363236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 762/1514] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:34:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:34:08.369431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa6317d-5fd3-45f4-b2b5-bbbdce2cd2a3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:08.369836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa6317d-5fd3-45f4-b2b5-bbbdce2cd2a3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:08.369992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa6317d-5fd3-45f4-b2b5-bbbdce2cd2a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:08.389635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-caa6317d-5fd3-45f4-b2b5-bbbdce2cd2a3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:34:08.390836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-caa6317d-5fd3-45f4-b2b5-bbbdce2cd2a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:08.398579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 753/1515] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:34:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:34:08.550469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2675d15c-ec2d-439e-88f2-f314247a2fe4 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:34:08.565523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2675d15c-ec2d-439e-88f2-f314247a2fe4 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:34:08.571042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c545af9-57cb-4d40-b810-abd5538547ff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:34:08.571633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c545af9-57cb-4d40-b810-abd5538547ff service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:08.571915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c545af9-57cb-4d40-b810-abd5538547ff service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:08.572190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c545af9-57cb-4d40-b810-abd5538547ff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:08.580622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c545af9-57cb-4d40-b810-abd5538547ff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:34:08.582201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c545af9-57cb-4d40-b810-abd5538547ff service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:34:08.591904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 763/1516] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:34:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:34:08.754826 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c0c4d-79f3-4ec0-b3a5-c0d9dc9e1730 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:08.755116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c0c4d-79f3-4ec0-b3a5-c0d9dc9e1730 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:08.755402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c0c4d-79f3-4ec0-b3a5-c0d9dc9e1730 None None] PATH_INFO: `/v3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:08.756739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 754/1517] 10.222.0.52 () {54 vars in 925 bytes} [Mon Sep 26 01:34:08 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:34:08.763625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208cb3a1-b31c-44c1-bb07-1a7de73e084c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:08.763912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208cb3a1-b31c-44c1-bb07-1a7de73e084c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:08.764190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208cb3a1-b31c-44c1-bb07-1a7de73e084c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:08.823476 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-208cb3a1-b31c-44c1-bb07-1a7de73e084c None None] MFA Rules not processed for user `427016f8da514dc4bbed21de08207dbb`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:34:08.824886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-208cb3a1-b31c-44c1-bb07-1a7de73e084c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:08.842289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:34:08.896086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 764/1518] 10.222.0.52 () {58 vars in 1015 bytes} [Mon Sep 26 01:34:08 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:34:12.478254 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b33405-aa95-4a11-95bd-d882394ab7db None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:12.478732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b33405-aa95-4a11-95bd-d882394ab7db None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:12.478846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b33405-aa95-4a11-95bd-d882394ab7db None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:12.480148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 755/1519] 10.222.0.25 () {54 vars in 961 bytes} [Mon Sep 26 01:34:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:34:12.485992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77a900b-8aaf-4175-b17b-7428fd06d68b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:12.486431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77a900b-8aaf-4175-b17b-7428fd06d68b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:12.486540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77a900b-8aaf-4175-b17b-7428fd06d68b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:12.505823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b77a900b-8aaf-4175-b17b-7428fd06d68b None None] MFA Rules not processed for user `427016f8da514dc4bbed21de08207dbb`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:34:12.507034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b77a900b-8aaf-4175-b17b-7428fd06d68b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:12.515911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 765/1520] 10.222.0.25 () {58 vars in 1058 bytes} [Mon Sep 26 01:34:12 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:34:12.523397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b33405-aa95-4a11-95bd-d882394ab7db None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:34:12.537299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b33405-aa95-4a11-95bd-d882394ab7db None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:34:12.542244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cfce43-3068-4efa-8942-5ffea61b992f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:34:12.542827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cfce43-3068-4efa-8942-5ffea61b992f service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:12.543131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cfce43-3068-4efa-8942-5ffea61b992f service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:12.543404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cfce43-3068-4efa-8942-5ffea61b992f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:12.552229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4cfce43-3068-4efa-8942-5ffea61b992f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:34:12.553909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4cfce43-3068-4efa-8942-5ffea61b992f service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:34:12.563566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 756/1521] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:34:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:34:20.073962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc67e70f-54ea-42e0-9a3e-e04dc8a69712 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:20.074453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc67e70f-54ea-42e0-9a3e-e04dc8a69712 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:20.074558 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc67e70f-54ea-42e0-9a3e-e04dc8a69712 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:20.076627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 766/1522] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:34:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:34:20.082783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f278c1-a918-48e1-8e31-1c4246f4db3b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:20.083234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f278c1-a918-48e1-8e31-1c4246f4db3b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:20.083353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f278c1-a918-48e1-8e31-1c4246f4db3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:20.103061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-94f278c1-a918-48e1-8e31-1c4246f4db3b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:34:20.104270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-94f278c1-a918-48e1-8e31-1c4246f4db3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:20.112854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 757/1523] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:34:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:34:20.266139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc67e70f-54ea-42e0-9a3e-e04dc8a69712 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:34:20.281816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc67e70f-54ea-42e0-9a3e-e04dc8a69712 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:34:20.287406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebab8e5-00b8-4d56-90f1-613efb8276a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:34:20.287948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebab8e5-00b8-4d56-90f1-613efb8276a8 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:20.288224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebab8e5-00b8-4d56-90f1-613efb8276a8 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:20.288535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebab8e5-00b8-4d56-90f1-613efb8276a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:20.296954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebab8e5-00b8-4d56-90f1-613efb8276a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:34:20.298534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebab8e5-00b8-4d56-90f1-613efb8276a8 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:34:20.307749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 767/1524] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:34:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:34:21.381471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ce59bb-2177-4e2e-9256-208279ef0542 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:21.382007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ce59bb-2177-4e2e-9256-208279ef0542 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:21.382007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ce59bb-2177-4e2e-9256-208279ef0542 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:21.383282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 758/1525] 127.0.0.1 () {54 vars in 901 bytes} [Mon Sep 26 01:34:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:34:21.390089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba79c6-140c-41a8-ab92-dcc5025948fb None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:21.390530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba79c6-140c-41a8-ab92-dcc5025948fb None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:21.390690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba79c6-140c-41a8-ab92-dcc5025948fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:21.412550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-8cba79c6-140c-41a8-ab92-dcc5025948fb None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:34:21.413756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8cba79c6-140c-41a8-ab92-dcc5025948fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:21.422280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 768/1526] 127.0.0.1 () {58 vars in 998 bytes} [Mon Sep 26 01:34:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:34:21.573643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ce59bb-2177-4e2e-9256-208279ef0542 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:34:21.588513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ce59bb-2177-4e2e-9256-208279ef0542 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:34:21.594568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683a315c-dcfd-4c23-9b53-fc52645077cb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:34:21.595150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683a315c-dcfd-4c23-9b53-fc52645077cb service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:21.595424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683a315c-dcfd-4c23-9b53-fc52645077cb service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:21.595695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683a315c-dcfd-4c23-9b53-fc52645077cb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:21.603642 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683a315c-dcfd-4c23-9b53-fc52645077cb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:34:21.605226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683a315c-dcfd-4c23-9b53-fc52645077cb service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:34:21.614985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 759/1527] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:34:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:34:25.702267 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a7b623-bb82-4b69-9fb2-620acf4fa08a None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:25.702844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a7b623-bb82-4b69-9fb2-620acf4fa08a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:25.702844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a7b623-bb82-4b69-9fb2-620acf4fa08a None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:25.704117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 769/1528] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:34:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:34:25.709301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff14832-9328-44a9-aae4-e79f73b2177d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:25.709703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff14832-9328-44a9-aae4-e79f73b2177d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:25.709859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff14832-9328-44a9-aae4-e79f73b2177d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:25.729602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-eff14832-9328-44a9-aae4-e79f73b2177d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:34:25.730809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-eff14832-9328-44a9-aae4-e79f73b2177d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:25.739050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 760/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:34:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:34:26.030464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a7b623-bb82-4b69-9fb2-620acf4fa08a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:34:26.032188 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-19a7b623-bb82-4b69-9fb2-620acf4fa08a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:26.113807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a7b623-bb82-4b69-9fb2-620acf4fa08a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:34:26.117370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b61d6e-7b01-48b2-bee1-c891a328090d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:34:26.117971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b61d6e-7b01-48b2-bee1-c891a328090d service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:26.118248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b61d6e-7b01-48b2-bee1-c891a328090d service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:26.118521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b61d6e-7b01-48b2-bee1-c891a328090d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:26.127851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b61d6e-7b01-48b2-bee1-c891a328090d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:34:26.129290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b61d6e-7b01-48b2-bee1-c891a328090d service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:34:26.139317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 770/1530] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:34:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:34:28.295872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccd548a-14cf-4c82-b16f-c5ff04d4790f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:28.296433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccd548a-14cf-4c82-b16f-c5ff04d4790f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:28.296433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccd548a-14cf-4c82-b16f-c5ff04d4790f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:28.297725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 761/1531] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:34:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:34:28.302812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2779fb03-ef52-4898-9f4d-fa9e21e612cb None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:28.303213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2779fb03-ef52-4898-9f4d-fa9e21e612cb None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:28.303351 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2779fb03-ef52-4898-9f4d-fa9e21e612cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:28.322907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-2779fb03-ef52-4898-9f4d-fa9e21e612cb None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:34:28.324104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2779fb03-ef52-4898-9f4d-fa9e21e612cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:28.332683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 771/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:34:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:34:28.621695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccd548a-14cf-4c82-b16f-c5ff04d4790f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:34:28.627720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=75 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:34:28.634024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccd548a-14cf-4c82-b16f-c5ff04d4790f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:34:28.637579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a50f46f-f173-4b2c-b2c9-59c93211715a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:34:28.638473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a50f46f-f173-4b2c-b2c9-59c93211715a service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:28.638792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a50f46f-f173-4b2c-b2c9-59c93211715a service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:28.639068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a50f46f-f173-4b2c-b2c9-59c93211715a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:28.647699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a50f46f-f173-4b2c-b2c9-59c93211715a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:34:28.649177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a50f46f-f173-4b2c-b2c9-59c93211715a service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:34:28.658506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 762/1533] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:34:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:34:32.278232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9dfc0c-b7d2-4862-8ed6-7dcb1d8d9c84 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:32.278980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9dfc0c-b7d2-4862-8ed6-7dcb1d8d9c84 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:32.278980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9dfc0c-b7d2-4862-8ed6-7dcb1d8d9c84 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:32.280154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 772/1534] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:34:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:34:32.285583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64383fc-e250-486c-9e28-d0a19354a0e8 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:32.286031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64383fc-e250-486c-9e28-d0a19354a0e8 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:32.286162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64383fc-e250-486c-9e28-d0a19354a0e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:32.306233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a64383fc-e250-486c-9e28-d0a19354a0e8 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:34:32.307615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a64383fc-e250-486c-9e28-d0a19354a0e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:32.316219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 763/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:34:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:34:32.620327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9dfc0c-b7d2-4862-8ed6-7dcb1d8d9c84 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:34:32.622587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8c9dfc0c-b7d2-4862-8ed6-7dcb1d8d9c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:32.687230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:34:32.701217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9dfc0c-b7d2-4862-8ed6-7dcb1d8d9c84 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:34:32.704807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d44f9ca-32ef-4695-ba09-1e452059dc8b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:34:32.705404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d44f9ca-32ef-4695-ba09-1e452059dc8b service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:32.705682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d44f9ca-32ef-4695-ba09-1e452059dc8b service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:32.705953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d44f9ca-32ef-4695-ba09-1e452059dc8b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:32.714661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d44f9ca-32ef-4695-ba09-1e452059dc8b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:34:32.716650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d44f9ca-32ef-4695-ba09-1e452059dc8b service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:34:32.725880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 773/1536] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:34:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:34:34.882078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ecdd76-afbc-409c-a07d-403593edf13d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:34.882675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ecdd76-afbc-409c-a07d-403593edf13d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:34.882675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ecdd76-afbc-409c-a07d-403593edf13d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:34.884006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 764/1537] 127.0.0.1 () {54 vars in 917 bytes} [Mon Sep 26 01:34:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:34:34.890027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1c2414-1cf4-4b38-956f-6eb10d28ff23 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:34.890440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1c2414-1cf4-4b38-956f-6eb10d28ff23 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:34.890578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1c2414-1cf4-4b38-956f-6eb10d28ff23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:34.910090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7e1c2414-1cf4-4b38-956f-6eb10d28ff23 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:34:34.911335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7e1c2414-1cf4-4b38-956f-6eb10d28ff23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:34.919341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 774/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Mon Sep 26 01:34:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:34:35.226238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ecdd76-afbc-409c-a07d-403593edf13d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:34:35.227933 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e1ecdd76-afbc-409c-a07d-403593edf13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:34:35.241772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ecdd76-afbc-409c-a07d-403593edf13d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:34:35.245338 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21482e12-5018-43df-b10d-671cdceddebe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:34:35.245926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21482e12-5018-43df-b10d-671cdceddebe service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:34:35.246203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21482e12-5018-43df-b10d-671cdceddebe service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:34:35.246475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21482e12-5018-43df-b10d-671cdceddebe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:34:35.256036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21482e12-5018-43df-b10d-671cdceddebe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:34:35.257482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21482e12-5018-43df-b10d-671cdceddebe service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:34:35.266832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 765/1539] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:34:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:34:44.905207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=11 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:34:45.262765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=14 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:01.085804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ce34e7-cc13-4c7e-af22-0fff268ff155 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.088814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ce34e7-cc13-4c7e-af22-0fff268ff155 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.088814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ce34e7-cc13-4c7e-af22-0fff268ff155 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.122789 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-50ce34e7-cc13-4c7e-af22-0fff268ff155 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:01.123992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-50ce34e7-cc13-4c7e-af22-0fff268ff155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:01.134782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 775/1540] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:01.145246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bde83b0-7098-4ad0-b649-ba5e62bf5d5f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.147607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bde83b0-7098-4ad0-b649-ba5e62bf5d5f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.147607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bde83b0-7098-4ad0-b649-ba5e62bf5d5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.177427 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7bde83b0-7098-4ad0-b649-ba5e62bf5d5f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:01.178857 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7bde83b0-7098-4ad0-b649-ba5e62bf5d5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:01.192755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 766/1541] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:01.197364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ce34e7-cc13-4c7e-af22-0fff268ff155 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:01.208212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ce34e7-cc13-4c7e-af22-0fff268ff155 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:01.215594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb6f40d-b993-4549-8db9-6110083702b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:01.216216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb6f40d-b993-4549-8db9-6110083702b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.216538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb6f40d-b993-4549-8db9-6110083702b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.216812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb6f40d-b993-4549-8db9-6110083702b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.217650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb6f40d-b993-4549-8db9-6110083702b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:01.217958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb6f40d-b993-4549-8db9-6110083702b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:01.219580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb6f40d-b993-4549-8db9-6110083702b8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:01.234660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 776/1542] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:01.246174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bde83b0-7098-4ad0-b649-ba5e62bf5d5f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:01.256155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bde83b0-7098-4ad0-b649-ba5e62bf5d5f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:01.265941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3828a690-cf1e-4a6a-af68-28de7e60a4f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:01.266560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828a690-cf1e-4a6a-af68-28de7e60a4f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.266838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828a690-cf1e-4a6a-af68-28de7e60a4f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.267108 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828a690-cf1e-4a6a-af68-28de7e60a4f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.267757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3828a690-cf1e-4a6a-af68-28de7e60a4f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:01.269709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3828a690-cf1e-4a6a-af68-28de7e60a4f1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:01.338457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 767/1543] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:01 2022] POST /identity/v3/projects => generated 385 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:01.344055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb6f40d-b993-4549-8db9-6110083702b8 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:01.362175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb6f40d-b993-4549-8db9-6110083702b8 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:01.466560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69f3998-d3c6-4ac3-8e7d-da890504830c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:01.467182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f3998-d3c6-4ac3-8e7d-da890504830c admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.467460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f3998-d3c6-4ac3-8e7d-da890504830c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.467731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f3998-d3c6-4ac3-8e7d-da890504830c admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.468384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69f3998-d3c6-4ac3-8e7d-da890504830c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:01.470318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69f3998-d3c6-4ac3-8e7d-da890504830c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:01.503411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 777/1544] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:01 2022] POST /identity/v3/users => generated 293 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:01.506964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3828a690-cf1e-4a6a-af68-28de7e60a4f1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:01.521566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3828a690-cf1e-4a6a-af68-28de7e60a4f1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:01.531906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1079f28-5cdc-4757-83a7-73d100d0ed0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:01.532653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1079f28-5cdc-4757-83a7-73d100d0ed0f admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.532934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1079f28-5cdc-4757-83a7-73d100d0ed0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.533206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1079f28-5cdc-4757-83a7-73d100d0ed0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.533870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1079f28-5cdc-4757-83a7-73d100d0ed0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:01.535554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1079f28-5cdc-4757-83a7-73d100d0ed0f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:01.554535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 768/1545] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:01 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:01.559118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69f3998-d3c6-4ac3-8e7d-da890504830c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:01.572271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69f3998-d3c6-4ac3-8e7d-da890504830c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:01.581254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ce9c61-0e50-4686-b1bf-e69bbfeb0e0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:01.581893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ce9c61-0e50-4686-b1bf-e69bbfeb0e0e admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.582175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ce9c61-0e50-4686-b1bf-e69bbfeb0e0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.582450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ce9c61-0e50-4686-b1bf-e69bbfeb0e0e admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.583306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ce9c61-0e50-4686-b1bf-e69bbfeb0e0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:01.584979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ce9c61-0e50-4686-b1bf-e69bbfeb0e0e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:01.599611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 778/1546] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:01 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:01.600755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1079f28-5cdc-4757-83a7-73d100d0ed0f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:01.615024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1079f28-5cdc-4757-83a7-73d100d0ed0f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:01.625207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88bd4af-a9c1-4345-9109-e708b5d33140 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:01.625995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88bd4af-a9c1-4345-9109-e708b5d33140 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.626277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88bd4af-a9c1-4345-9109-e708b5d33140 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.626548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88bd4af-a9c1-4345-9109-e708b5d33140 admin admin] PATH_INFO: `/v3/projects/f7cff87a197d4ddfbaeb0ac3d6d87fa9/users/b6f18267454a4d148dd701ecb1109aac/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.651690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88bd4af-a9c1-4345-9109-e708b5d33140 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7cff87a197d4ddfbaeb0ac3d6d87fa9, user_id=b6f18267454a4d148dd701ecb1109aac, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:01.651931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88bd4af-a9c1-4345-9109-e708b5d33140 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:01.677678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 769/1547] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:01 2022] PUT /identity/v3/projects/f7cff87a197d4ddfbaeb0ac3d6d87fa9/users/b6f18267454a4d148dd701ecb1109aac/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:01.696154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d731088d-350f-47c9-93ea-dcf4d62a2d5d None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.696635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d731088d-350f-47c9-93ea-dcf4d62a2d5d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.696915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d731088d-350f-47c9-93ea-dcf4d62a2d5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.745883 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-d731088d-350f-47c9-93ea-dcf4d62a2d5d None None] MFA Rules not processed for user `b6f18267454a4d148dd701ecb1109aac`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:01.747338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d731088d-350f-47c9-93ea-dcf4d62a2d5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:01.751158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32203da3-eaba-4961-91c6-cb5e9d823396 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.751781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32203da3-eaba-4961-91c6-cb5e9d823396 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.752071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32203da3-eaba-4961-91c6-cb5e9d823396 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.785961 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-32203da3-eaba-4961-91c6-cb5e9d823396 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:01.787225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-32203da3-eaba-4961-91c6-cb5e9d823396 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:01.874520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 779/1548] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:01.876309 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6608887-f7dd-4cb8-8ffd-07f418e9652a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.876913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6608887-f7dd-4cb8-8ffd-07f418e9652a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.877418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6608887-f7dd-4cb8-8ffd-07f418e9652a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.934175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b6608887-f7dd-4cb8-8ffd-07f418e9652a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:01.935725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b6608887-f7dd-4cb8-8ffd-07f418e9652a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:01.935969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 770/1549] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:01.939767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102432e6-ac5b-458b-9828-5f60e785d81c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.939955 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102432e6-ac5b-458b-9828-5f60e785d81c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.940225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102432e6-ac5b-458b-9828-5f60e785d81c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.959126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 780/1550] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:01.961591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9630aaa-370f-4c38-b2f3-688e991ff057 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:01.961875 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9630aaa-370f-4c38-b2f3-688e991ff057 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:01.962145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9630aaa-370f-4c38-b2f3-688e991ff057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:01.991058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-d9630aaa-370f-4c38-b2f3-688e991ff057 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:01.996372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d9630aaa-370f-4c38-b2f3-688e991ff057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:01.997645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-102432e6-ac5b-458b-9828-5f60e785d81c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:01.998309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-102432e6-ac5b-458b-9828-5f60e785d81c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.010599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 781/1551] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.010599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f364c-bf2b-49f1-8dc1-ff086d8e4993 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.010599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f364c-bf2b-49f1-8dc1-ff086d8e4993 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.010599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f364c-bf2b-49f1-8dc1-ff086d8e4993 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.018329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 782/1552] 10.222.0.25 () {54 vars in 961 bytes} [Mon Sep 26 01:36:02 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:02.019655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee090bdb-7bc4-4a25-bcc4-1746bb58c6d5 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.020222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee090bdb-7bc4-4a25-bcc4-1746bb58c6d5 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.020222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee090bdb-7bc4-4a25-bcc4-1746bb58c6d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.028626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 771/1553] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.028626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73fdcc8-080d-499e-aea2-2f9aae0d80cc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.028626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73fdcc8-080d-499e-aea2-2f9aae0d80cc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.028626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73fdcc8-080d-499e-aea2-2f9aae0d80cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.076043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-ee090bdb-7bc4-4a25-bcc4-1746bb58c6d5 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.077321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ee090bdb-7bc4-4a25-bcc4-1746bb58c6d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.078633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d73fdcc8-080d-499e-aea2-2f9aae0d80cc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.079796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d73fdcc8-080d-499e-aea2-2f9aae0d80cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.085785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 783/1554] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.087262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc064c-bac4-4b7f-af9e-aea60b479731 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.087262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc064c-bac4-4b7f-af9e-aea60b479731 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.087849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc064c-bac4-4b7f-af9e-aea60b479731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.092077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 772/1555] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.092077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01962b34-5f29-46e9-990d-28b4192d0991 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.092077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01962b34-5f29-46e9-990d-28b4192d0991 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.092658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01962b34-5f29-46e9-990d-28b4192d0991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.116179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-01962b34-5f29-46e9-990d-28b4192d0991 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.117657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-01962b34-5f29-46e9-990d-28b4192d0991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.130301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 773/1556] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.131547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a0abaf-b096-46c6-9bed-446790012a74 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.131955 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a0abaf-b096-46c6-9bed-446790012a74 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.132235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a0abaf-b096-46c6-9bed-446790012a74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.136904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-76fc064c-bac4-4b7f-af9e-aea60b479731 None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.138160 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-76fc064c-bac4-4b7f-af9e-aea60b479731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.147871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:02.153870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-91a0abaf-b096-46c6-9bed-446790012a74 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.155059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-91a0abaf-b096-46c6-9bed-446790012a74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.162948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 774/1557] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.164199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87007e7a-1587-4b6a-9851-513744a7aa12 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.164520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87007e7a-1587-4b6a-9851-513744a7aa12 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.164794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87007e7a-1587-4b6a-9851-513744a7aa12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.184054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-87007e7a-1587-4b6a-9851-513744a7aa12 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.185281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-87007e7a-1587-4b6a-9851-513744a7aa12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.194307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 775/1558] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.195609 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88981a06-bf89-472f-8405-a67441d628d9 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.195931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88981a06-bf89-472f-8405-a67441d628d9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.196196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88981a06-bf89-472f-8405-a67441d628d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.219483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-88981a06-bf89-472f-8405-a67441d628d9 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.220959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-88981a06-bf89-472f-8405-a67441d628d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.222241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 784/1559] 10.222.0.25 () {58 vars in 1058 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.223088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fc064c-bac4-4b7f-af9e-aea60b479731 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.229893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 776/1560] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.230889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88981a06-bf89-472f-8405-a67441d628d9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.237523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fc064c-bac4-4b7f-af9e-aea60b479731 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.241379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88981a06-bf89-472f-8405-a67441d628d9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.242192 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd57fa68-6847-4468-a40f-04ad86aaf8cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.242796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd57fa68-6847-4468-a40f-04ad86aaf8cd admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.243075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd57fa68-6847-4468-a40f-04ad86aaf8cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.243350 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd57fa68-6847-4468-a40f-04ad86aaf8cd admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.244027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd57fa68-6847-4468-a40f-04ad86aaf8cd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:02.244333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd57fa68-6847-4468-a40f-04ad86aaf8cd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.246047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd57fa68-6847-4468-a40f-04ad86aaf8cd admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.246552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6d9ed3-9adf-4a3a-85db-46625db8fd95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.247174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6d9ed3-9adf-4a3a-85db-46625db8fd95 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.247454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6d9ed3-9adf-4a3a-85db-46625db8fd95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.247722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6d9ed3-9adf-4a3a-85db-46625db8fd95 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.248379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6d9ed3-9adf-4a3a-85db-46625db8fd95 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:02.248989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6d9ed3-9adf-4a3a-85db-46625db8fd95 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.249747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-bb6d9ed3-9adf-4a3a-85db-46625db8fd95 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:36:02.249747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 26 01:36:02.249747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: Sep 26 01:36:02.250649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6d9ed3-9adf-4a3a-85db-46625db8fd95 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.258168 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 785/1561] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:02.270340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 777/1562] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:02.270340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6d9ed3-9adf-4a3a-85db-46625db8fd95 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.270636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27c6e4-f0f9-473b-a36d-93737faf08b8 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.270636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27c6e4-f0f9-473b-a36d-93737faf08b8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.270636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27c6e4-f0f9-473b-a36d-93737faf08b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.274957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6d9ed3-9adf-4a3a-85db-46625db8fd95 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.280479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a2c198-4d75-4a0a-a15a-3912c400d045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.281070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a2c198-4d75-4a0a-a15a-3912c400d045 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.281348 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a2c198-4d75-4a0a-a15a-3912c400d045 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.281618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a2c198-4d75-4a0a-a15a-3912c400d045 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.282247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a2c198-4d75-4a0a-a15a-3912c400d045 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:02.282549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a2c198-4d75-4a0a-a15a-3912c400d045 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.284176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a2c198-4d75-4a0a-a15a-3912c400d045 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.287624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-fa27c6e4-f0f9-473b-a36d-93737faf08b8 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.288864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fa27c6e4-f0f9-473b-a36d-93737faf08b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.297247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 778/1563] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:02.297425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 786/1564] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.297760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a2c198-4d75-4a0a-a15a-3912c400d045 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.298434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e54488-836a-411a-83ab-dd710161e303 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.298722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e54488-836a-411a-83ab-dd710161e303 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.298997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e54488-836a-411a-83ab-dd710161e303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.317250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a2c198-4d75-4a0a-a15a-3912c400d045 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.322207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2b0d8b-8f64-46de-9920-42d682e9f4a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.322395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-f1e54488-836a-411a-83ab-dd710161e303 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.323505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b0d8b-8f64-46de-9920-42d682e9f4a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.324003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b0d8b-8f64-46de-9920-42d682e9f4a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.324241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f1e54488-836a-411a-83ab-dd710161e303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.324644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b0d8b-8f64-46de-9920-42d682e9f4a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.325272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2b0d8b-8f64-46de-9920-42d682e9f4a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:02.325576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2b0d8b-8f64-46de-9920-42d682e9f4a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.327205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2b0d8b-8f64-46de-9920-42d682e9f4a3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.332578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 787/1565] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.333428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e54488-836a-411a-83ab-dd710161e303 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.350497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 779/1566] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:02.350497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c5b861-9dbd-4f84-ba82-05aa5e726684 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.350497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c5b861-9dbd-4f84-ba82-05aa5e726684 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.350497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c5b861-9dbd-4f84-ba82-05aa5e726684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.350967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e54488-836a-411a-83ab-dd710161e303 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.352263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4cf163-e4a4-4f31-ba50-e810cac329a9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.352935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4cf163-e4a4-4f31-ba50-e810cac329a9 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.353217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4cf163-e4a4-4f31-ba50-e810cac329a9 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.353489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4cf163-e4a4-4f31-ba50-e810cac329a9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.361769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4cf163-e4a4-4f31-ba50-e810cac329a9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.363357 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4cf163-e4a4-4f31-ba50-e810cac329a9 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.364780 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-97c5b861-9dbd-4f84-ba82-05aa5e726684 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.365969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-97c5b861-9dbd-4f84-ba82-05aa5e726684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.372266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 788/1567] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:02 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:02.373054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4cf163-e4a4-4f31-ba50-e810cac329a9 service nova] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.378189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 780/1568] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.379035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c5b861-9dbd-4f84-ba82-05aa5e726684 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.383048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4cf163-e4a4-4f31-ba50-e810cac329a9 service nova] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.384336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:02.389325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0bef18-6cf9-4de5-a9d8-af084decd9f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.389991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0bef18-6cf9-4de5-a9d8-af084decd9f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.390244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c5b861-9dbd-4f84-ba82-05aa5e726684 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.390413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0bef18-6cf9-4de5-a9d8-af084decd9f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.390793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0bef18-6cf9-4de5-a9d8-af084decd9f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.391523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0bef18-6cf9-4de5-a9d8-af084decd9f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.392671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: WARNING py.warnings [None req-5b0bef18-6cf9-4de5-a9d8-af084decd9f3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:36:02.392671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: warnings.warn(msg) Sep 26 01:36:02.392671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: Sep 26 01:36:02.394282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0bef18-6cf9-4de5-a9d8-af084decd9f3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.396740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53675cf-2406-434d-beb8-8427de30502f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.397787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53675cf-2406-434d-beb8-8427de30502f admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.398068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53675cf-2406-434d-beb8-8427de30502f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.398337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53675cf-2406-434d-beb8-8427de30502f admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.398958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53675cf-2406-434d-beb8-8427de30502f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.401545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53675cf-2406-434d-beb8-8427de30502f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.427368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 789/1569] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/projects => generated 375 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:02.428574 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0bef18-6cf9-4de5-a9d8-af084decd9f3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.432523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 781/1570] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/projects => generated 391 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:02.433940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646aadb0-2639-4fbf-88bc-63d765616ddb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.434336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646aadb0-2639-4fbf-88bc-63d765616ddb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.434721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646aadb0-2639-4fbf-88bc-63d765616ddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.454780 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-646aadb0-2639-4fbf-88bc-63d765616ddb None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:02.456134 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-646aadb0-2639-4fbf-88bc-63d765616ddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:02.507960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0bef18-6cf9-4de5-a9d8-af084decd9f3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.519990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 782/1571] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:02.521171 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646aadb0-2639-4fbf-88bc-63d765616ddb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.530999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646aadb0-2639-4fbf-88bc-63d765616ddb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.535886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0e7006-afcd-4d2b-8dfc-3c7b18eca29c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.536499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0e7006-afcd-4d2b-8dfc-3c7b18eca29c admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.536781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0e7006-afcd-4d2b-8dfc-3c7b18eca29c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.537054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0e7006-afcd-4d2b-8dfc-3c7b18eca29c admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.537695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0e7006-afcd-4d2b-8dfc-3c7b18eca29c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:02.537999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0e7006-afcd-4d2b-8dfc-3c7b18eca29c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.539592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0e7006-afcd-4d2b-8dfc-3c7b18eca29c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.551365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 783/1572] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:02.552138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0e7006-afcd-4d2b-8dfc-3c7b18eca29c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.562837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0e7006-afcd-4d2b-8dfc-3c7b18eca29c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.567464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e56af64-9678-48a0-b3db-f02197ec7e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.568079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e56af64-9678-48a0-b3db-f02197ec7e06 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.568314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:02.568486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e56af64-9678-48a0-b3db-f02197ec7e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.568646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e56af64-9678-48a0-b3db-f02197ec7e06 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.569344 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e56af64-9678-48a0-b3db-f02197ec7e06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.571175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e56af64-9678-48a0-b3db-f02197ec7e06 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.581437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e527890-e841-401c-be7d-be2151f30c2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.582349 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e527890-e841-401c-be7d-be2151f30c2a admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.582790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e527890-e841-401c-be7d-be2151f30c2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.583216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e527890-e841-401c-be7d-be2151f30c2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.583992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e527890-e841-401c-be7d-be2151f30c2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.586102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e527890-e841-401c-be7d-be2151f30c2a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.603956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 784/1573] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/projects => generated 379 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:02.603956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e56af64-9678-48a0-b3db-f02197ec7e06 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.621672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 790/1574] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/projects => generated 387 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:02.622804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e527890-e841-401c-be7d-be2151f30c2a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.668876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e56af64-9678-48a0-b3db-f02197ec7e06 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.687443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e527890-e841-401c-be7d-be2151f30c2a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.692547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0be634-2194-491b-86e7-27ef7a1c0b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.693534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0be634-2194-491b-86e7-27ef7a1c0b88 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.693820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0be634-2194-491b-86e7-27ef7a1c0b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.694090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0be634-2194-491b-86e7-27ef7a1c0b88 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.694716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0be634-2194-491b-86e7-27ef7a1c0b88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.697200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0be634-2194-491b-86e7-27ef7a1c0b88 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.718647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:02.720078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 791/1575] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/users => generated 288 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:02.720829 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0be634-2194-491b-86e7-27ef7a1c0b88 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.730684 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09b1b63-f718-472c-b003-e1957e206623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.730926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0be634-2194-491b-86e7-27ef7a1c0b88 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.731493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09b1b63-f718-472c-b003-e1957e206623 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.731855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09b1b63-f718-472c-b003-e1957e206623 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.732138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09b1b63-f718-472c-b003-e1957e206623 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.732833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09b1b63-f718-472c-b003-e1957e206623 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:02.733134 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09b1b63-f718-472c-b003-e1957e206623 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.734795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09b1b63-f718-472c-b003-e1957e206623 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.736517 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf30e1-fe72-4914-8607-76c95811f255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.737421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf30e1-fe72-4914-8607-76c95811f255 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.737421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf30e1-fe72-4914-8607-76c95811f255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.737659 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf30e1-fe72-4914-8607-76c95811f255 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.738266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cf30e1-fe72-4914-8607-76c95811f255 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.740237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cf30e1-fe72-4914-8607-76c95811f255 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.747496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 785/1576] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:02.749486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09b1b63-f718-472c-b003-e1957e206623 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.761268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09b1b63-f718-472c-b003-e1957e206623 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.761746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 792/1577] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/users => generated 296 bytes in 41 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:02.762615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf30e1-fe72-4914-8607-76c95811f255 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.766168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a0d595-e98f-4492-8f4a-d14d4b30b3e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.766757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a0d595-e98f-4492-8f4a-d14d4b30b3e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.767034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a0d595-e98f-4492-8f4a-d14d4b30b3e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.767301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a0d595-e98f-4492-8f4a-d14d4b30b3e6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.768136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a0d595-e98f-4492-8f4a-d14d4b30b3e6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:02.768473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a0d595-e98f-4492-8f4a-d14d4b30b3e6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.770630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a0d595-e98f-4492-8f4a-d14d4b30b3e6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.772304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf30e1-fe72-4914-8607-76c95811f255 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.777833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d410e9-9307-42e9-b801-40ea5c292aee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.778436 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d410e9-9307-42e9-b801-40ea5c292aee admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.778700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d410e9-9307-42e9-b801-40ea5c292aee admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.778970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d410e9-9307-42e9-b801-40ea5c292aee admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.779592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d410e9-9307-42e9-b801-40ea5c292aee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.781514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d410e9-9307-42e9-b801-40ea5c292aee admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.783897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 786/1578] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:02.791806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a0d595-e98f-4492-8f4a-d14d4b30b3e6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.802961 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a0d595-e98f-4492-8f4a-d14d4b30b3e6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.808524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b9b118-bcd5-4b5a-aa1a-2d009b30cdbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.809150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b9b118-bcd5-4b5a-aa1a-2d009b30cdbb admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.809497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b9b118-bcd5-4b5a-aa1a-2d009b30cdbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.809742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b9b118-bcd5-4b5a-aa1a-2d009b30cdbb admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.810367 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b9b118-bcd5-4b5a-aa1a-2d009b30cdbb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.811371 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-b0b9b118-bcd5-4b5a-aa1a-2d009b30cdbb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:36:02.811371 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 26 01:36:02.811371 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: Sep 26 01:36:02.815480 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 793/1579] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/projects => generated 405 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:02.815640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b9b118-bcd5-4b5a-aa1a-2d009b30cdbb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.815741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d410e9-9307-42e9-b801-40ea5c292aee admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.839843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 787/1580] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/users => generated 290 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:02.839843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3195883a-9987-4c84-b94f-5bc963b1ae47 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.839843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3195883a-9987-4c84-b94f-5bc963b1ae47 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.839843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3195883a-9987-4c84-b94f-5bc963b1ae47 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.840432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 788/1581] 10.222.0.25 () {54 vars in 912 bytes} [Mon Sep 26 01:36:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:02.841335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3195883a-9987-4c84-b94f-5bc963b1ae47 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.881374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d410e9-9307-42e9-b801-40ea5c292aee admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.886284 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3003b0-a99c-4b16-94fb-878877d70926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.886893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3003b0-a99c-4b16-94fb-878877d70926 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.887171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3003b0-a99c-4b16-94fb-878877d70926 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.887449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3003b0-a99c-4b16-94fb-878877d70926 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.888066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3003b0-a99c-4b16-94fb-878877d70926 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.890025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3003b0-a99c-4b16-94fb-878877d70926 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.904747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3195883a-9987-4c84-b94f-5bc963b1ae47 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.909523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3bb14d-5980-4359-a106-99527798fd04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.910181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3bb14d-5980-4359-a106-99527798fd04 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.910620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3bb14d-5980-4359-a106-99527798fd04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.910871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3bb14d-5980-4359-a106-99527798fd04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.910994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 794/1582] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/users => generated 294 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:02.911943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3003b0-a99c-4b16-94fb-878877d70926 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.912923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3bb14d-5980-4359-a106-99527798fd04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.914623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3bb14d-5980-4359-a106-99527798fd04 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.923010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3003b0-a99c-4b16-94fb-878877d70926 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.925414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 789/1583] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:02 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:02.926229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3bb14d-5980-4359-a106-99527798fd04 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.930512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d1e05a-737b-4112-a685-f9ed3ed6fbe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.931106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d1e05a-737b-4112-a685-f9ed3ed6fbe5 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.931383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d1e05a-737b-4112-a685-f9ed3ed6fbe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.931656 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d1e05a-737b-4112-a685-f9ed3ed6fbe5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.932624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d1e05a-737b-4112-a685-f9ed3ed6fbe5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.934438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d1e05a-737b-4112-a685-f9ed3ed6fbe5 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.937272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3bb14d-5980-4359-a106-99527798fd04 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:02.941809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bbc62e-075b-445d-8300-1efd4c795c0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:02.942389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bbc62e-075b-445d-8300-1efd4c795c0d admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:02.942670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bbc62e-075b-445d-8300-1efd4c795c0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:02.942948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bbc62e-075b-445d-8300-1efd4c795c0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:02.943582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bbc62e-075b-445d-8300-1efd4c795c0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:02.945191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bbc62e-075b-445d-8300-1efd4c795c0d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:02.951214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=88,keystone:INSERT=12 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:02.957578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 790/1584] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:02.958603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bbc62e-075b-445d-8300-1efd4c795c0d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.966665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 795/1585] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/projects => generated 405 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:02.967633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d1e05a-737b-4112-a685-f9ed3ed6fbe5 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:02.968486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bbc62e-075b-445d-8300-1efd4c795c0d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.026235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bdae1d-1dac-4731-a713-c6009f5b0400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.027014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bdae1d-1dac-4731-a713-c6009f5b0400 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.027300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bdae1d-1dac-4731-a713-c6009f5b0400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.027570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bdae1d-1dac-4731-a713-c6009f5b0400 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.028203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bdae1d-1dac-4731-a713-c6009f5b0400 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.030131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bdae1d-1dac-4731-a713-c6009f5b0400 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.031049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d1e05a-737b-4112-a685-f9ed3ed6fbe5 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.035716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323e5611-b4ae-4943-8dd1-331c00a2892d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.036318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323e5611-b4ae-4943-8dd1-331c00a2892d admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.036736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323e5611-b4ae-4943-8dd1-331c00a2892d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.037019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323e5611-b4ae-4943-8dd1-331c00a2892d admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.037647 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323e5611-b4ae-4943-8dd1-331c00a2892d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.039600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323e5611-b4ae-4943-8dd1-331c00a2892d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.059432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 791/1586] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/projects => generated 387 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:03.060637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bdae1d-1dac-4731-a713-c6009f5b0400 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.063705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 796/1587] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:02 2022] POST /identity/v3/users => generated 303 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:03.063705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a10eae-1a1a-491f-ae8d-3eaeed4b7335 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.063705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a10eae-1a1a-491f-ae8d-3eaeed4b7335 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.063705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a10eae-1a1a-491f-ae8d-3eaeed4b7335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.083139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-a0a10eae-1a1a-491f-ae8d-3eaeed4b7335 None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:03.084357 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a0a10eae-1a1a-491f-ae8d-3eaeed4b7335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:03.094025 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:03.127524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bdae1d-1dac-4731-a713-c6009f5b0400 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.132538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222b8aae-a434-41c1-bbf7-c68be1435c24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.133182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222b8aae-a434-41c1-bbf7-c68be1435c24 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.133486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222b8aae-a434-41c1-bbf7-c68be1435c24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.133734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222b8aae-a434-41c1-bbf7-c68be1435c24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.134378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222b8aae-a434-41c1-bbf7-c68be1435c24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.136634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222b8aae-a434-41c1-bbf7-c68be1435c24 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.146370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 792/1588] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.147261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222b8aae-a434-41c1-bbf7-c68be1435c24 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.153484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 797/1589] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:36:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:03.154264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a10eae-1a1a-491f-ae8d-3eaeed4b7335 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.157878 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222b8aae-a434-41c1-bbf7-c68be1435c24 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.163647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d094b0a-30be-45e8-9444-f2566a360b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.164278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a10eae-1a1a-491f-ae8d-3eaeed4b7335 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.164445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d094b0a-30be-45e8-9444-f2566a360b7e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.164828 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d094b0a-30be-45e8-9444-f2566a360b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.165177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d094b0a-30be-45e8-9444-f2566a360b7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.165890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d094b0a-30be-45e8-9444-f2566a360b7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.168528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d094b0a-30be-45e8-9444-f2566a360b7e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.169904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591b1417-f5e8-456f-a973-2eab671ceeda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.170495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b1417-f5e8-456f-a973-2eab671ceeda admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.170776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b1417-f5e8-456f-a973-2eab671ceeda admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.171053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b1417-f5e8-456f-a973-2eab671ceeda admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.171695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591b1417-f5e8-456f-a973-2eab671ceeda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.173367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591b1417-f5e8-456f-a973-2eab671ceeda admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.179404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 793/1590] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.180325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d094b0a-30be-45e8-9444-f2566a360b7e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.184324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 798/1591] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.185145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591b1417-f5e8-456f-a973-2eab671ceeda admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.193253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d094b0a-30be-45e8-9444-f2566a360b7e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.195203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591b1417-f5e8-456f-a973-2eab671ceeda admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.197805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa73d61-d667-4e23-895e-a323a0ffda1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.198481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa73d61-d667-4e23-895e-a323a0ffda1d admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.198759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa73d61-d667-4e23-895e-a323a0ffda1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.199600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa73d61-d667-4e23-895e-a323a0ffda1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.200042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa73d61-d667-4e23-895e-a323a0ffda1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.200321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f42c57c-6847-4a15-9af5-cca91636714b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.200992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f42c57c-6847-4a15-9af5-cca91636714b admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.201269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f42c57c-6847-4a15-9af5-cca91636714b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.201539 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f42c57c-6847-4a15-9af5-cca91636714b admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.201644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa73d61-d667-4e23-895e-a323a0ffda1d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.202152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f42c57c-6847-4a15-9af5-cca91636714b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.204044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f42c57c-6847-4a15-9af5-cca91636714b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.211481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 794/1592] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.212241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa73d61-d667-4e23-895e-a323a0ffda1d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.222425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa73d61-d667-4e23-895e-a323a0ffda1d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.226942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a2c4ef-e75b-485b-875b-5b9a32deef65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.231097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 799/1593] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:03 2022] POST /identity/v3/users => generated 303 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:03.231220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a2c4ef-e75b-485b-875b-5b9a32deef65 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.231220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a2c4ef-e75b-485b-875b-5b9a32deef65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.231426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f42c57c-6847-4a15-9af5-cca91636714b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.231516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a2c4ef-e75b-485b-875b-5b9a32deef65 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.231516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a2c4ef-e75b-485b-875b-5b9a32deef65 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.231516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a2c4ef-e75b-485b-875b-5b9a32deef65 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.241333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f42c57c-6847-4a15-9af5-cca91636714b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.246715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f9da3-1152-41f3-bd1b-d4a4c1d7c872 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.247314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f9da3-1152-41f3-bd1b-d4a4c1d7c872 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.247586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f9da3-1152-41f3-bd1b-d4a4c1d7c872 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.247856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f9da3-1152-41f3-bd1b-d4a4c1d7c872 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.248523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79f9da3-1152-41f3-bd1b-d4a4c1d7c872 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.250109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79f9da3-1152-41f3-bd1b-d4a4c1d7c872 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.252490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 795/1594] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:03 2022] POST /identity/v3/users => generated 294 bytes in 40 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:03.253447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a2c4ef-e75b-485b-875b-5b9a32deef65 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.263044 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a2c4ef-e75b-485b-875b-5b9a32deef65 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.264394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 800/1595] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.265248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f9da3-1152-41f3-bd1b-d4a4c1d7c872 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.270487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9537d5-61cc-4097-8291-cd007281b4ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.271077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9537d5-61cc-4097-8291-cd007281b4ff admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.271358 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9537d5-61cc-4097-8291-cd007281b4ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.271630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9537d5-61cc-4097-8291-cd007281b4ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.272299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9537d5-61cc-4097-8291-cd007281b4ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.273937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9537d5-61cc-4097-8291-cd007281b4ff admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.285136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 796/1596] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.286002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9537d5-61cc-4097-8291-cd007281b4ff admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.295763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9537d5-61cc-4097-8291-cd007281b4ff admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.300329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b791f30-a5a1-425e-bb93-8a47178f076e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.300927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b791f30-a5a1-425e-bb93-8a47178f076e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.301203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b791f30-a5a1-425e-bb93-8a47178f076e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.301478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b791f30-a5a1-425e-bb93-8a47178f076e admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.302108 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b791f30-a5a1-425e-bb93-8a47178f076e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.303792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b791f30-a5a1-425e-bb93-8a47178f076e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.311190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:03.314599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 797/1597] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.315778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b791f30-a5a1-425e-bb93-8a47178f076e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.326798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b791f30-a5a1-425e-bb93-8a47178f076e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.331374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee111a53-ae0c-44bb-b08f-458c84339a56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.332016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee111a53-ae0c-44bb-b08f-458c84339a56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.332295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee111a53-ae0c-44bb-b08f-458c84339a56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.332424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f9da3-1152-41f3-bd1b-d4a4c1d7c872 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.332600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee111a53-ae0c-44bb-b08f-458c84339a56 admin admin] PATH_INFO: `/v3/projects/3677555d86804f03ae243765cfd3fd24/users/06043e862a394764a58003f205cbb484/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.337101 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13baf35-9d7a-4c47-bfa4-2660bd7666fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.337710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13baf35-9d7a-4c47-bfa4-2660bd7666fe service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.337993 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13baf35-9d7a-4c47-bfa4-2660bd7666fe service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.338378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13baf35-9d7a-4c47-bfa4-2660bd7666fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.347966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee111a53-ae0c-44bb-b08f-458c84339a56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3677555d86804f03ae243765cfd3fd24, user_id=06043e862a394764a58003f205cbb484, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.348816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13baf35-9d7a-4c47-bfa4-2660bd7666fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.350249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13baf35-9d7a-4c47-bfa4-2660bd7666fe service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.350699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee111a53-ae0c-44bb-b08f-458c84339a56 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.361768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 801/1598] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:03.362565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13baf35-9d7a-4c47-bfa4-2660bd7666fe service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.365221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 798/1599] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:03 2022] PUT /identity/v3/projects/3677555d86804f03ae243765cfd3fd24/users/06043e862a394764a58003f205cbb484/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:03.365848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee111a53-ae0c-44bb-b08f-458c84339a56 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.375367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13baf35-9d7a-4c47-bfa4-2660bd7666fe service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.435556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1e1de7-766a-44d3-949e-1929f08fe6ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.436285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1e1de7-766a-44d3-949e-1929f08fe6ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.436712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1e1de7-766a-44d3-949e-1929f08fe6ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.436997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1e1de7-766a-44d3-949e-1929f08fe6ad admin admin] PATH_INFO: `/v3/projects/a553921e97994322b71fda7d032ed8bf/users/fb54ba1355004e14b8dc43cf3c028e67/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.438156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee111a53-ae0c-44bb-b08f-458c84339a56 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.446708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa353ee-7f97-4aa0-852e-b061abe08214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.448108 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa353ee-7f97-4aa0-852e-b061abe08214 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.448491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa353ee-7f97-4aa0-852e-b061abe08214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.449000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa353ee-7f97-4aa0-852e-b061abe08214 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.449977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa353ee-7f97-4aa0-852e-b061abe08214 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.451668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa353ee-7f97-4aa0-852e-b061abe08214 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.453864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1e1de7-766a-44d3-949e-1929f08fe6ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=a553921e97994322b71fda7d032ed8bf, user_id=fb54ba1355004e14b8dc43cf3c028e67, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.455095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: WARNING py.warnings [None req-fe1e1de7-766a-44d3-949e-1929f08fe6ad admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:36:03.455095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: warnings.warn(msg) Sep 26 01:36:03.455095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: Sep 26 01:36:03.457156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1e1de7-766a-44d3-949e-1929f08fe6ad admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.463926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 799/1600] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.465275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa353ee-7f97-4aa0-852e-b061abe08214 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.473114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 802/1601] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:03 2022] PUT /identity/v3/projects/a553921e97994322b71fda7d032ed8bf/users/fb54ba1355004e14b8dc43cf3c028e67/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:03.474148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1e1de7-766a-44d3-949e-1929f08fe6ad admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.480888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa353ee-7f97-4aa0-852e-b061abe08214 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.515847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:03.542667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1e1de7-766a-44d3-949e-1929f08fe6ad admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.547551 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc738c64-cda1-4791-9612-fe2339c06b2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.548333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc738c64-cda1-4791-9612-fe2339c06b2e admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.548649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc738c64-cda1-4791-9612-fe2339c06b2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.548965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc738c64-cda1-4791-9612-fe2339c06b2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.549662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc738c64-cda1-4791-9612-fe2339c06b2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.550577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ebb74c-d4f0-4f88-bce3-676f1e4136c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.551221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ebb74c-d4f0-4f88-bce3-676f1e4136c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.551459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc738c64-cda1-4791-9612-fe2339c06b2e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.551571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ebb74c-d4f0-4f88-bce3-676f1e4136c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.551757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ebb74c-d4f0-4f88-bce3-676f1e4136c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.552434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ebb74c-d4f0-4f88-bce3-676f1e4136c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.554036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ebb74c-d4f0-4f88-bce3-676f1e4136c2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.562782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 803/1602] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.563804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc738c64-cda1-4791-9612-fe2339c06b2e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.564683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 800/1603] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.565483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ebb74c-d4f0-4f88-bce3-676f1e4136c2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.577737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc738c64-cda1-4791-9612-fe2339c06b2e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.577918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ebb74c-d4f0-4f88-bce3-676f1e4136c2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.582362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28be9f72-7d9c-4695-8fad-7b0a532cdfd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.583118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28be9f72-7d9c-4695-8fad-7b0a532cdfd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.583302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28be9f72-7d9c-4695-8fad-7b0a532cdfd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.583525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28be9f72-7d9c-4695-8fad-7b0a532cdfd9 admin admin] PATH_INFO: `/v3/projects/a9406983aa784d9ca3a16ddc6bffeb92/users/3787c797be334618b82824f022e390cc/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.583648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704af4a6-fdb2-4fbb-8d49-b54db5d874bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.584260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704af4a6-fdb2-4fbb-8d49-b54db5d874bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.584545 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704af4a6-fdb2-4fbb-8d49-b54db5d874bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.584801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704af4a6-fdb2-4fbb-8d49-b54db5d874bb admin admin] PATH_INFO: `/v3/projects/089c1c6e9cb94a19b451d0da50ba9bea/users/73c365df91254de6a9217dab0280eff3/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.600747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28be9f72-7d9c-4695-8fad-7b0a532cdfd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9406983aa784d9ca3a16ddc6bffeb92, user_id=3787c797be334618b82824f022e390cc, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.601915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704af4a6-fdb2-4fbb-8d49-b54db5d874bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=089c1c6e9cb94a19b451d0da50ba9bea, user_id=73c365df91254de6a9217dab0280eff3, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.603590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28be9f72-7d9c-4695-8fad-7b0a532cdfd9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.605021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704af4a6-fdb2-4fbb-8d49-b54db5d874bb admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.620766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 801/1604] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:03 2022] PUT /identity/v3/projects/a9406983aa784d9ca3a16ddc6bffeb92/users/3787c797be334618b82824f022e390cc/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:03.624635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 804/1605] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:03 2022] PUT /identity/v3/projects/089c1c6e9cb94a19b451d0da50ba9bea/users/73c365df91254de6a9217dab0280eff3/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:03.624635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704af4a6-fdb2-4fbb-8d49-b54db5d874bb admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.625864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4392803d-c0f5-40f1-80b4-ad4acd68672b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.626208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4392803d-c0f5-40f1-80b4-ad4acd68672b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.628721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4392803d-c0f5-40f1-80b4-ad4acd68672b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.652900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4392803d-c0f5-40f1-80b4-ad4acd68672b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:03.654385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4392803d-c0f5-40f1-80b4-ad4acd68672b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:03.670237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:03.695394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704af4a6-fdb2-4fbb-8d49-b54db5d874bb admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.700270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947a580f-3104-4aa7-8914-3ae1cff177c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.700913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947a580f-3104-4aa7-8914-3ae1cff177c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.701177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947a580f-3104-4aa7-8914-3ae1cff177c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.701593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947a580f-3104-4aa7-8914-3ae1cff177c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.702313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947a580f-3104-4aa7-8914-3ae1cff177c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.703879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947a580f-3104-4aa7-8914-3ae1cff177c3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.714138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 805/1606] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.715178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43de5d47-9e25-4eeb-9bd2-bee057f1891c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.715470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43de5d47-9e25-4eeb-9bd2-bee057f1891c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.715747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43de5d47-9e25-4eeb-9bd2-bee057f1891c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.727635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 802/1607] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:03.728629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4392803d-c0f5-40f1-80b4-ad4acd68672b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.738194 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-43de5d47-9e25-4eeb-9bd2-bee057f1891c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:03.740109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4392803d-c0f5-40f1-80b4-ad4acd68672b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.740790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-43de5d47-9e25-4eeb-9bd2-bee057f1891c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:03.744940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99644bc6-58ff-4ea7-b120-572b294c9a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.745581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99644bc6-58ff-4ea7-b120-572b294c9a00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.745895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99644bc6-58ff-4ea7-b120-572b294c9a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.746158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99644bc6-58ff-4ea7-b120-572b294c9a00 admin admin] PATH_INFO: `/v3/projects/670e7230a6e94ef99dfd035d3b37676b/users/691549d6715444459307c2583bd824de/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.752700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 806/1608] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:03.752700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43de5d47-9e25-4eeb-9bd2-bee057f1891c None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.760724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43de5d47-9e25-4eeb-9bd2-bee057f1891c None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.767406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cc8246-f9a6-4d89-8284-757f842f2521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.768022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cc8246-f9a6-4d89-8284-757f842f2521 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.768134 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99644bc6-58ff-4ea7-b120-572b294c9a00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=670e7230a6e94ef99dfd035d3b37676b, user_id=691549d6715444459307c2583bd824de, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.768582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cc8246-f9a6-4d89-8284-757f842f2521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.768865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cc8246-f9a6-4d89-8284-757f842f2521 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.769619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cc8246-f9a6-4d89-8284-757f842f2521 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.771094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99644bc6-58ff-4ea7-b120-572b294c9a00 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.771279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cc8246-f9a6-4d89-8284-757f842f2521 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.782689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 807/1609] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:03.784268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128962bd-8569-4de7-91bd-5ab4af4da949 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.784731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128962bd-8569-4de7-91bd-5ab4af4da949 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.785112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128962bd-8569-4de7-91bd-5ab4af4da949 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.793587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 803/1610] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:03 2022] PUT /identity/v3/projects/670e7230a6e94ef99dfd035d3b37676b/users/691549d6715444459307c2583bd824de/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:03.793587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc5f316-1eee-42f0-a446-8cdcc0c6e1ba None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.793587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc5f316-1eee-42f0-a446-8cdcc0c6e1ba None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.793587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc5f316-1eee-42f0-a446-8cdcc0c6e1ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.813182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-128962bd-8569-4de7-91bd-5ab4af4da949 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:03.814526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-128962bd-8569-4de7-91bd-5ab4af4da949 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:03.823024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4dc5f316-1eee-42f0-a446-8cdcc0c6e1ba None None] MFA Rules not processed for user `3787c797be334618b82824f022e390cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:03.823978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4dc5f316-1eee-42f0-a446-8cdcc0c6e1ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:03.900706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 808/1611] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:03.901459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128962bd-8569-4de7-91bd-5ab4af4da949 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.912701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128962bd-8569-4de7-91bd-5ab4af4da949 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:03.913159 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 804/1612] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:03 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:03.914231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc5f316-1eee-42f0-a446-8cdcc0c6e1ba None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:03.923432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33591ba4-20c0-43bd-b3e1-a009ddc353bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:03.924938 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33591ba4-20c0-43bd-b3e1-a009ddc353bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:03.925445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33591ba4-20c0-43bd-b3e1-a009ddc353bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:03.926487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33591ba4-20c0-43bd-b3e1-a009ddc353bd admin admin] PATH_INFO: `/v3/projects/f6ae179ea71e41a79a5858d177458a1e/users/8b7f2ed50e784c34bd0e8db5245facac/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:03.929003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:03.946094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33591ba4-20c0-43bd-b3e1-a009ddc353bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6ae179ea71e41a79a5858d177458a1e, user_id=8b7f2ed50e784c34bd0e8db5245facac, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:03.949102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33591ba4-20c0-43bd-b3e1-a009ddc353bd admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:03.974235 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 809/1613] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:03 2022] PUT /identity/v3/projects/f6ae179ea71e41a79a5858d177458a1e/users/8b7f2ed50e784c34bd0e8db5245facac/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:03.975085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33591ba4-20c0-43bd-b3e1-a009ddc353bd admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.010160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc5f316-1eee-42f0-a446-8cdcc0c6e1ba None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.061419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33591ba4-20c0-43bd-b3e1-a009ddc353bd admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.065796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe380af5-205c-4649-966c-22bf215a4e66 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.066575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe380af5-205c-4649-966c-22bf215a4e66 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.066979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe380af5-205c-4649-966c-22bf215a4e66 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.067387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe380af5-205c-4649-966c-22bf215a4e66 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.096814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9e963d-a9b2-494a-ab8f-d009bce9cab5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.097440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e963d-a9b2-494a-ab8f-d009bce9cab5 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.097708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e963d-a9b2-494a-ab8f-d009bce9cab5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.097985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e963d-a9b2-494a-ab8f-d009bce9cab5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.120322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:04.139667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe380af5-205c-4649-966c-22bf215a4e66 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.141197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe380af5-205c-4649-966c-22bf215a4e66 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.151763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 810/1614] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:04.152771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe380af5-205c-4649-966c-22bf215a4e66 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.162092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe380af5-205c-4649-966c-22bf215a4e66 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.167362 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8194ae-e1d5-426e-8b03-828b60a21aaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.168020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8194ae-e1d5-426e-8b03-828b60a21aaa admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.168299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8194ae-e1d5-426e-8b03-828b60a21aaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.168606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8194ae-e1d5-426e-8b03-828b60a21aaa admin admin] PATH_INFO: `/v3/projects/2f78ca6e22f04c4ab911b2e308338517/users/bc62747b8aab40d9bff9d853b9b73199/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.186397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9e963d-a9b2-494a-ab8f-d009bce9cab5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.188068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8194ae-e1d5-426e-8b03-828b60a21aaa admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f78ca6e22f04c4ab911b2e308338517, user_id=bc62747b8aab40d9bff9d853b9b73199, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.189562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9e963d-a9b2-494a-ab8f-d009bce9cab5 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.191468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8194ae-e1d5-426e-8b03-828b60a21aaa admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.205128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 805/1615] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:04.206422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8d066c-ec0b-4758-bec4-fe5c5d48703c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.206809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 811/1616] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:04 2022] PUT /identity/v3/projects/2f78ca6e22f04c4ab911b2e308338517/users/bc62747b8aab40d9bff9d853b9b73199/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:04.206936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8d066c-ec0b-4758-bec4-fe5c5d48703c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.207037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8d066c-ec0b-4758-bec4-fe5c5d48703c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.211449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8194ae-e1d5-426e-8b03-828b60a21aaa admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.213876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7c8194ae-e1d5-426e-8b03-828b60a21aaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:04.249947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2d8d066c-ec0b-4758-bec4-fe5c5d48703c None None] MFA Rules not processed for user `691549d6715444459307c2583bd824de`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:04.251697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2d8d066c-ec0b-4758-bec4-fe5c5d48703c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:04.310718 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8194ae-e1d5-426e-8b03-828b60a21aaa admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.315061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa63380-20e9-4dd8-9064-997540d6bcb6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.315713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa63380-20e9-4dd8-9064-997540d6bcb6 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.315923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa63380-20e9-4dd8-9064-997540d6bcb6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.316196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa63380-20e9-4dd8-9064-997540d6bcb6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.341197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 806/1617] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:04 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:04.344884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8d066c-ec0b-4758-bec4-fe5c5d48703c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.353401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:04.409133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa63380-20e9-4dd8-9064-997540d6bcb6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.410602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa63380-20e9-4dd8-9064-997540d6bcb6 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.422754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 812/1618] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:04.424226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55070876-3458-4eaa-8961-e2b147bb2723 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.424629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55070876-3458-4eaa-8961-e2b147bb2723 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.424976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55070876-3458-4eaa-8961-e2b147bb2723 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.430207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8d066c-ec0b-4758-bec4-fe5c5d48703c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.439002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4c31a1-c8cc-485b-8b47-1742a53626a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.440169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c31a1-c8cc-485b-8b47-1742a53626a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.440557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c31a1-c8cc-485b-8b47-1742a53626a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.441442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c31a1-c8cc-485b-8b47-1742a53626a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.442379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4c31a1-c8cc-485b-8b47-1742a53626a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.445128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4c31a1-c8cc-485b-8b47-1742a53626a8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.446627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-55070876-3458-4eaa-8961-e2b147bb2723 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:04.448578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-55070876-3458-4eaa-8961-e2b147bb2723 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:04.458573 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 813/1619] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:04.460144 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1c36f4-4dda-4c60-9b0e-56c99aa225a1 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.460556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1c36f4-4dda-4c60-9b0e-56c99aa225a1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.460867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1c36f4-4dda-4c60-9b0e-56c99aa225a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.486333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-3e1c36f4-4dda-4c60-9b0e-56c99aa225a1 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:04.488183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3e1c36f4-4dda-4c60-9b0e-56c99aa225a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:04.489086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 807/1620] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:04 2022] POST /identity/v3/projects => generated 385 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:04.494819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02d6d99-4501-466c-af2b-1bbef5256bc3 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.494819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02d6d99-4501-466c-af2b-1bbef5256bc3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.494819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02d6d99-4501-466c-af2b-1bbef5256bc3 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.496772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 808/1621] 10.222.0.25 () {54 vars in 961 bytes} [Mon Sep 26 01:36:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:04.497640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02d6d99-4501-466c-af2b-1bbef5256bc3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.580702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:04.596808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 814/1622] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:04.597704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1c36f4-4dda-4c60-9b0e-56c99aa225a1 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.622626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02d6d99-4501-466c-af2b-1bbef5256bc3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.629372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86980b01-9ed0-4dc3-94e2-a13436c964d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.630264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86980b01-9ed0-4dc3-94e2-a13436c964d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.631015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86980b01-9ed0-4dc3-94e2-a13436c964d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.631426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86980b01-9ed0-4dc3-94e2-a13436c964d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.632202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86980b01-9ed0-4dc3-94e2-a13436c964d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.635686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86980b01-9ed0-4dc3-94e2-a13436c964d9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.673576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1c36f4-4dda-4c60-9b0e-56c99aa225a1 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.678129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd0b45c-06e6-4ff1-990a-3d06ef89e4b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.678940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd0b45c-06e6-4ff1-990a-3d06ef89e4b5 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.679156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 809/1623] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:04 2022] POST /identity/v3/projects => generated 401 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:04.679248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd0b45c-06e6-4ff1-990a-3d06ef89e4b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.679492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd0b45c-06e6-4ff1-990a-3d06ef89e4b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.679998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86980b01-9ed0-4dc3-94e2-a13436c964d9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.693830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd0b45c-06e6-4ff1-990a-3d06ef89e4b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.696601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd0b45c-06e6-4ff1-990a-3d06ef89e4b5 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.711761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 815/1624] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:04.714061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9948c5-1995-4cf4-8c32-2dc66021c2d0 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.714443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9948c5-1995-4cf4-8c32-2dc66021c2d0 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.714945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9948c5-1995-4cf4-8c32-2dc66021c2d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.741017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-8c9948c5-1995-4cf4-8c32-2dc66021c2d0 None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:04.742506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8c9948c5-1995-4cf4-8c32-2dc66021c2d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:04.808022 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86980b01-9ed0-4dc3-94e2-a13436c964d9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.821352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f786b6f3-6426-4098-8dc9-4a3798f5fc54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.823282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f786b6f3-6426-4098-8dc9-4a3798f5fc54 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.823568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f786b6f3-6426-4098-8dc9-4a3798f5fc54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.825989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f786b6f3-6426-4098-8dc9-4a3798f5fc54 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.827178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f786b6f3-6426-4098-8dc9-4a3798f5fc54 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.832590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f786b6f3-6426-4098-8dc9-4a3798f5fc54 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.839767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 816/1625] 10.222.0.25 () {58 vars in 1058 bytes} [Mon Sep 26 01:36:04 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:04.840777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9948c5-1995-4cf4-8c32-2dc66021c2d0 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.878035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 810/1626] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:04 2022] POST /identity/v3/users => generated 299 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:04.879124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f786b6f3-6426-4098-8dc9-4a3798f5fc54 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.894003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f786b6f3-6426-4098-8dc9-4a3798f5fc54 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.899301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a9bc58-daba-4567-96b5-a5f4b315eaf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.900213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a9bc58-daba-4567-96b5-a5f4b315eaf2 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.901867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a9bc58-daba-4567-96b5-a5f4b315eaf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.901994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a9bc58-daba-4567-96b5-a5f4b315eaf2 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.902725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a9bc58-daba-4567-96b5-a5f4b315eaf2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.904905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a9bc58-daba-4567-96b5-a5f4b315eaf2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.915073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:04.924544 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9948c5-1995-4cf4-8c32-2dc66021c2d0 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.931262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc8e936-9415-4178-838a-33c011373dab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.931471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc8e936-9415-4178-838a-33c011373dab service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.932039 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc8e936-9415-4178-838a-33c011373dab service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.932377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc8e936-9415-4178-838a-33c011373dab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.937269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 811/1627] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:04 2022] POST /identity/v3/users => generated 307 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:04.939089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47e8ff5-7c69-47a4-aafc-e006247d9bb2 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.939714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47e8ff5-7c69-47a4-aafc-e006247d9bb2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.940198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47e8ff5-7c69-47a4-aafc-e006247d9bb2 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.942610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 812/1628] 10.222.0.25 () {54 vars in 912 bytes} [Mon Sep 26 01:36:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:04.943850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47e8ff5-7c69-47a4-aafc-e006247d9bb2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.949211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc8e936-9415-4178-838a-33c011373dab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.951120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc8e936-9415-4178-838a-33c011373dab service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.964026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47e8ff5-7c69-47a4-aafc-e006247d9bb2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.964406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 817/1629] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:04.965208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc8e936-9415-4178-838a-33c011373dab service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:04.971150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc9e273-6f98-4ef9-a4a2-541e0739b2a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.972812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc9e273-6f98-4ef9-a4a2-541e0739b2a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.973586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc9e273-6f98-4ef9-a4a2-541e0739b2a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.974146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc9e273-6f98-4ef9-a4a2-541e0739b2a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.975407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc9e273-6f98-4ef9-a4a2-541e0739b2a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.975837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc8e936-9415-4178-838a-33c011373dab service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:04.979746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc9e273-6f98-4ef9-a4a2-541e0739b2a8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.983420 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34cac25-5f7b-4df0-b05c-20fea811ff9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:04.983825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34cac25-5f7b-4df0-b05c-20fea811ff9b admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.984179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34cac25-5f7b-4df0-b05c-20fea811ff9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.985404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34cac25-5f7b-4df0-b05c-20fea811ff9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:04.986159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34cac25-5f7b-4df0-b05c-20fea811ff9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:04.988570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34cac25-5f7b-4df0-b05c-20fea811ff9b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:04.996325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 813/1630] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:04 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:04.998242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da67561-057a-4100-8fd8-5524304d0f3e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:04.998695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da67561-057a-4100-8fd8-5524304d0f3e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:04.999212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da67561-057a-4100-8fd8-5524304d0f3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:05.008200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 818/1631] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:04 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:05.010044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34cac25-5f7b-4df0-b05c-20fea811ff9b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:05.024152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34cac25-5f7b-4df0-b05c-20fea811ff9b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:05.027671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4da67561-057a-4100-8fd8-5524304d0f3e None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:05.029133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4da67561-057a-4100-8fd8-5524304d0f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:05.040091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 814/1632] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:36:04 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:05.041419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da67561-057a-4100-8fd8-5524304d0f3e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:05.053656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da67561-057a-4100-8fd8-5524304d0f3e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:05.060726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc7612-35a2-4701-ad6f-249b19feb172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:05.061739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc7612-35a2-4701-ad6f-249b19feb172 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:05.062115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc7612-35a2-4701-ad6f-249b19feb172 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:05.062543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc7612-35a2-4701-ad6f-249b19feb172 admin admin] PATH_INFO: `/v3/projects/c06fcc7562514314971e32566197db67/users/e0b83ab36c574e93bfb6ea01a5e0d667/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:05.066991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06cb853-fd10-460d-8422-1d10566f532a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:05.067672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06cb853-fd10-460d-8422-1d10566f532a admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:05.067954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06cb853-fd10-460d-8422-1d10566f532a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:05.069440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06cb853-fd10-460d-8422-1d10566f532a admin admin] PATH_INFO: `/v3/projects/964d347c1bf8455d81a115b3c805e622/users/a15972d85de84e369221497aa1715c80/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:05.084898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dc7612-35a2-4701-ad6f-249b19feb172 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c06fcc7562514314971e32566197db67, user_id=e0b83ab36c574e93bfb6ea01a5e0d667, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:05.089537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dc7612-35a2-4701-ad6f-249b19feb172 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:05.110007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 815/1633] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:05 2022] PUT /identity/v3/projects/c06fcc7562514314971e32566197db67/users/e0b83ab36c574e93bfb6ea01a5e0d667/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:05.111072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc7612-35a2-4701-ad6f-249b19feb172 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:05.115348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06cb853-fd10-460d-8422-1d10566f532a admin admin] RBAC: Authorizing `identity:create_grant(project_id=964d347c1bf8455d81a115b3c805e622, user_id=a15972d85de84e369221497aa1715c80, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:05.126641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06cb853-fd10-460d-8422-1d10566f532a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:05.156164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 819/1634] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:05 2022] PUT /identity/v3/projects/964d347c1bf8455d81a115b3c805e622/users/a15972d85de84e369221497aa1715c80/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:05.157209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06cb853-fd10-460d-8422-1d10566f532a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:05.173593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:05.191195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc7612-35a2-4701-ad6f-249b19feb172 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:05.264616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac696486-57ec-49d4-8e49-6216b2cd6258 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:05.265331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac696486-57ec-49d4-8e49-6216b2cd6258 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:05.265678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac696486-57ec-49d4-8e49-6216b2cd6258 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:05.266051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac696486-57ec-49d4-8e49-6216b2cd6258 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:05.321172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06cb853-fd10-460d-8422-1d10566f532a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:05.334899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f720cfc-1e17-4f86-90f1-a76e2fce7cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:05.335616 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f720cfc-1e17-4f86-90f1-a76e2fce7cfb admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:05.335898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f720cfc-1e17-4f86-90f1-a76e2fce7cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:05.336240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f720cfc-1e17-4f86-90f1-a76e2fce7cfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:05.337433 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f720cfc-1e17-4f86-90f1-a76e2fce7cfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:05.338980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f720cfc-1e17-4f86-90f1-a76e2fce7cfb admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:05.365006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac696486-57ec-49d4-8e49-6216b2cd6258 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:05.366674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac696486-57ec-49d4-8e49-6216b2cd6258 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:05.370190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 820/1635] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:05 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:05.370190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f720cfc-1e17-4f86-90f1-a76e2fce7cfb admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:05.379796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 816/1636] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 269 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:05.380836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac696486-57ec-49d4-8e49-6216b2cd6258 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:05.389884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:05.392960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f720cfc-1e17-4f86-90f1-a76e2fce7cfb admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:05.396219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac696486-57ec-49d4-8e49-6216b2cd6258 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:05.405062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a93311-ca69-4800-8c3a-2ef0c40cbe75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:05.405992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a93311-ca69-4800-8c3a-2ef0c40cbe75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:05.406293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a93311-ca69-4800-8c3a-2ef0c40cbe75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:05.406786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a93311-ca69-4800-8c3a-2ef0c40cbe75 admin admin] PATH_INFO: `/v3/projects/c06fcc7562514314971e32566197db67/users/e0b83ab36c574e93bfb6ea01a5e0d667/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:05.409992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a93311-ca69-4800-8c3a-2ef0c40cbe75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c06fcc7562514314971e32566197db67, user_id=e0b83ab36c574e93bfb6ea01a5e0d667, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:05.410292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b387ee3-03e1-4d41-8ff6-a0268e1803ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:05.410916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b387ee3-03e1-4d41-8ff6-a0268e1803ae admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:05.411198 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b387ee3-03e1-4d41-8ff6-a0268e1803ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:05.411474 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b387ee3-03e1-4d41-8ff6-a0268e1803ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:05.412125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b387ee3-03e1-4d41-8ff6-a0268e1803ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:05.413986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b387ee3-03e1-4d41-8ff6-a0268e1803ae admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:05.414276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a93311-ca69-4800-8c3a-2ef0c40cbe75 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:05.436301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 821/1638] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:05 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:05.437380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b387ee3-03e1-4d41-8ff6-a0268e1803ae admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:05.444276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 817/1638] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:05 2022] PUT /identity/v3/projects/c06fcc7562514314971e32566197db67/users/e0b83ab36c574e93bfb6ea01a5e0d667/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:05.446433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4324900-b523-4018-bd91-a92fc51f5547 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:05.446891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4324900-b523-4018-bd91-a92fc51f5547 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:05.447179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4324900-b523-4018-bd91-a92fc51f5547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:05.462007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b387ee3-03e1-4d41-8ff6-a0268e1803ae admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:05.527566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d4324900-b523-4018-bd91-a92fc51f5547 None None] MFA Rules not processed for user `e0b83ab36c574e93bfb6ea01a5e0d667`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:05.528835 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d4324900-b523-4018-bd91-a92fc51f5547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:05.590799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c4a5c7-5276-4083-bd4f-97c1538e27b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:05.594131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c4a5c7-5276-4083-bd4f-97c1538e27b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:05.594688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c4a5c7-5276-4083-bd4f-97c1538e27b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:05.595179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c4a5c7-5276-4083-bd4f-97c1538e27b6 admin admin] PATH_INFO: `/v3/projects/964d347c1bf8455d81a115b3c805e622/users/a15972d85de84e369221497aa1715c80/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:05.597923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c4a5c7-5276-4083-bd4f-97c1538e27b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=964d347c1bf8455d81a115b3c805e622, user_id=a15972d85de84e369221497aa1715c80, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:05.601092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c4a5c7-5276-4083-bd4f-97c1538e27b6 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:05.616821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 822/1639] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:05 2022] PUT /identity/v3/projects/964d347c1bf8455d81a115b3c805e622/users/a15972d85de84e369221497aa1715c80/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:05.629131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c506b8b7-8a8a-4ceb-9dc0-0cf8349c3159 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:05.633361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c506b8b7-8a8a-4ceb-9dc0-0cf8349c3159 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:05.634380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c506b8b7-8a8a-4ceb-9dc0-0cf8349c3159 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:05.688155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 818/1640] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:05 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:05.702817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4324900-b523-4018-bd91-a92fc51f5547 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:05.722244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-c506b8b7-8a8a-4ceb-9dc0-0cf8349c3159 None None] MFA Rules not processed for user `a15972d85de84e369221497aa1715c80`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:05.725277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c506b8b7-8a8a-4ceb-9dc0-0cf8349c3159 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:05.855532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4324900-b523-4018-bd91-a92fc51f5547 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:05.909761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 823/1641] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:05 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:05.951846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c6cc40-54be-4503-9e55-16d07f1d96cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:05.952568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6cc40-54be-4503-9e55-16d07f1d96cb service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:05.952851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6cc40-54be-4503-9e55-16d07f1d96cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:05.953123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6cc40-54be-4503-9e55-16d07f1d96cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:06.030820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:06.060702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c6cc40-54be-4503-9e55-16d07f1d96cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:06.073292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c6cc40-54be-4503-9e55-16d07f1d96cb service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:06.088905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 819/1642] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:05 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:06.116255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c506b8b7-8a8a-4ceb-9dc0-0cf8349c3159 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:06.130444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c506b8b7-8a8a-4ceb-9dc0-0cf8349c3159 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:06.144015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6320a4-ef6f-4831-90ec-ba9a340b65d2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:06.144397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6320a4-ef6f-4831-90ec-ba9a340b65d2 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:06.144728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6320a4-ef6f-4831-90ec-ba9a340b65d2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:06.146816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6320a4-ef6f-4831-90ec-ba9a340b65d2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:06.157284 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:06.165163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6320a4-ef6f-4831-90ec-ba9a340b65d2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:06.167308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6320a4-ef6f-4831-90ec-ba9a340b65d2 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:06.193486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 824/1643] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:06 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:07.269724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c6cc40-54be-4503-9e55-16d07f1d96cb service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:07.361917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c6cc40-54be-4503-9e55-16d07f1d96cb service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:07.376038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01460d49-8e4f-409e-8b45-78a3f5c94fcf None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:07.376632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01460d49-8e4f-409e-8b45-78a3f5c94fcf None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:07.376776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01460d49-8e4f-409e-8b45-78a3f5c94fcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:07.379269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:07.380093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:07.380522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:07.380873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] PATH_INFO: `/v3/users/3787c797be334618b82824f022e390cc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:07.382243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3787c797be334618b82824f022e390cc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:07.385585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:07.433459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-01460d49-8e4f-409e-8b45-78a3f5c94fcf None None] MFA Rules not processed for user `8b7f2ed50e784c34bd0e8db5245facac`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:07.437506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-01460d49-8e4f-409e-8b45-78a3f5c94fcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:07.536053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3787c797be334618b82824f022e390cc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:07.552554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3787c797be334618b82824f022e390cc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:07.555245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3787c797be334618b82824f022e390cc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:07.557065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3787c797be334618b82824f022e390cc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:07.586648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 825/1644] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:07 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:07.593715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01460d49-8e4f-409e-8b45-78a3f5c94fcf None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:07.596503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-64a1469b-77fd-4f4c-a614-6f3e05156fc6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3787c797be334618b82824f022e390cc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:07.598486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-01460d49-8e4f-409e-8b45-78a3f5c94fcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:07.620570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01460d49-8e4f-409e-8b45-78a3f5c94fcf None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:07.627713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddac775e-22a9-40e1-89f4-66edb7e18fe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:07.630678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddac775e-22a9-40e1-89f4-66edb7e18fe2 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:07.631181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddac775e-22a9-40e1-89f4-66edb7e18fe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:07.631525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddac775e-22a9-40e1-89f4-66edb7e18fe2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:07.632244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddac775e-22a9-40e1-89f4-66edb7e18fe2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:07.635415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddac775e-22a9-40e1-89f4-66edb7e18fe2 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:07.666598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 820/1645] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:07 2022] DELETE /identity/v3/users/3787c797be334618b82824f022e390cc => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:07.671600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ede71b7-b3a8-413e-98f1-52b2353e37a3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:07.672093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ede71b7-b3a8-413e-98f1-52b2353e37a3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:07.672524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ede71b7-b3a8-413e-98f1-52b2353e37a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:07.677906 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 826/1646] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:07 2022] POST /identity/v3/projects => generated 405 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:07.683461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddac775e-22a9-40e1-89f4-66edb7e18fe2 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:07.720609 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5ede71b7-b3a8-413e-98f1-52b2353e37a3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:07.723387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5ede71b7-b3a8-413e-98f1-52b2353e37a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:07.815939 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddac775e-22a9-40e1-89f4-66edb7e18fe2 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:07.825697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc1c45a-8305-46f1-b4c3-dd955e1f4121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:07.829303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc1c45a-8305-46f1-b4c3-dd955e1f4121 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:07.829899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc1c45a-8305-46f1-b4c3-dd955e1f4121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:07.831929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc1c45a-8305-46f1-b4c3-dd955e1f4121 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:07.832902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc1c45a-8305-46f1-b4c3-dd955e1f4121 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:07.837273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc1c45a-8305-46f1-b4c3-dd955e1f4121 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:07.850896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 821/1647] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:07.868818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ede71b7-b3a8-413e-98f1-52b2353e37a3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:07.872034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5ede71b7-b3a8-413e-98f1-52b2353e37a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:07.877547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 827/1648] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:07 2022] POST /identity/v3/users => generated 309 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:07.881046 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc1c45a-8305-46f1-b4c3-dd955e1f4121 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:07.881559 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:07.900837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc1c45a-8305-46f1-b4c3-dd955e1f4121 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:07.908137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835c62d3-8875-4dd8-92a4-c0433a701bd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:07.909449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835c62d3-8875-4dd8-92a4-c0433a701bd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:07.909885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835c62d3-8875-4dd8-92a4-c0433a701bd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:07.910492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835c62d3-8875-4dd8-92a4-c0433a701bd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:07.911356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835c62d3-8875-4dd8-92a4-c0433a701bd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:07.913450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835c62d3-8875-4dd8-92a4-c0433a701bd4 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:07.926971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 828/1649] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:07 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:36:07.931202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835c62d3-8875-4dd8-92a4-c0433a701bd4 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:07.991423 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835c62d3-8875-4dd8-92a4-c0433a701bd4 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:07.992262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea287301-caf1-4646-a922-5fc91864daba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:07.993041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea287301-caf1-4646-a922-5fc91864daba admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:07.993331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea287301-caf1-4646-a922-5fc91864daba admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:07.998405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea287301-caf1-4646-a922-5fc91864daba admin admin] PATH_INFO: `/v3/projects/977a345fcfff467f85889ab3d4a63aef/users/e99e0aa6af854e8aaa7c263c42814e77/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:08.038653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea287301-caf1-4646-a922-5fc91864daba admin admin] RBAC: Authorizing `identity:create_grant(project_id=977a345fcfff467f85889ab3d4a63aef, user_id=e99e0aa6af854e8aaa7c263c42814e77, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:08.044443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea287301-caf1-4646-a922-5fc91864daba admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:08.064720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 829/1650] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:07 2022] PUT /identity/v3/projects/977a345fcfff467f85889ab3d4a63aef/users/e99e0aa6af854e8aaa7c263c42814e77/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:08.068558 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea287301-caf1-4646-a922-5fc91864daba admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:08.075215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ede71b7-b3a8-413e-98f1-52b2353e37a3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:08.092043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:08.160716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c40da66-d839-47a6-8f8a-1d17dab05d10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:08.161406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea287301-caf1-4646-a922-5fc91864daba admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:08.161496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c40da66-d839-47a6-8f8a-1d17dab05d10 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:08.161868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c40da66-d839-47a6-8f8a-1d17dab05d10 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:08.162273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c40da66-d839-47a6-8f8a-1d17dab05d10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:08.170266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c9354d-5086-431f-9d4f-e906a2044c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:08.171090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c9354d-5086-431f-9d4f-e906a2044c00 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:08.171376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c9354d-5086-431f-9d4f-e906a2044c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:08.171657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c9354d-5086-431f-9d4f-e906a2044c00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:08.172407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c9354d-5086-431f-9d4f-e906a2044c00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:08.174091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c40da66-d839-47a6-8f8a-1d17dab05d10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:08.174292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c9354d-5086-431f-9d4f-e906a2044c00 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:08.175949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c40da66-d839-47a6-8f8a-1d17dab05d10 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:08.187767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 830/1651] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:08 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:36:08.191578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 822/1652] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:36:08.191830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c9354d-5086-431f-9d4f-e906a2044c00 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:08.214008 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c9354d-5086-431f-9d4f-e906a2044c00 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:08.224211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f6c537-d284-4665-94ea-4a74940a033f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:08.225240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f6c537-d284-4665-94ea-4a74940a033f admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:08.225623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f6c537-d284-4665-94ea-4a74940a033f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:08.226033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f6c537-d284-4665-94ea-4a74940a033f admin admin] PATH_INFO: `/v3/projects/977a345fcfff467f85889ab3d4a63aef/users/e99e0aa6af854e8aaa7c263c42814e77/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:08.228973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f6c537-d284-4665-94ea-4a74940a033f admin admin] RBAC: Authorizing `identity:create_grant(project_id=977a345fcfff467f85889ab3d4a63aef, user_id=e99e0aa6af854e8aaa7c263c42814e77, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:08.232202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f6c537-d284-4665-94ea-4a74940a033f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:08.253170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 831/1653] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:08 2022] PUT /identity/v3/projects/977a345fcfff467f85889ab3d4a63aef/users/e99e0aa6af854e8aaa7c263c42814e77/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:08.330862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c40da66-d839-47a6-8f8a-1d17dab05d10 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:08.333064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8c40da66-d839-47a6-8f8a-1d17dab05d10 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:08.348498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c40da66-d839-47a6-8f8a-1d17dab05d10 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:08.424967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:08.425612 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:08.425894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:08.426169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] PATH_INFO: `/v3/users/691549d6715444459307c2583bd824de` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:08.427711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] RBAC: Authorizing `identity:delete_user(user_id=691549d6715444459307c2583bd824de)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:08.430562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:08.505671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '691549d6715444459307c2583bd824de'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:08.522789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '691549d6715444459307c2583bd824de'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:08.524702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '691549d6715444459307c2583bd824de'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:08.527563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '691549d6715444459307c2583bd824de'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:08.545338 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '691549d6715444459307c2583bd824de'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:08.547896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:08.592701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 823/1654] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:08 2022] DELETE /identity/v3/users/691549d6715444459307c2583bd824de => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:08.596890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c3bb9-3726-4b5b-813f-6227430f6e93 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:08.597389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c3bb9-3726-4b5b-813f-6227430f6e93 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:08.597482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c3bb9-3726-4b5b-813f-6227430f6e93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:08.636264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e04c3bb9-3726-4b5b-813f-6227430f6e93 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:08.637553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e04c3bb9-3726-4b5b-813f-6227430f6e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:08.759448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 832/1655] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:08.813938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:08.815433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:09.032485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68d24a3-bdde-42ad-962a-837736e5b61c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:09.054856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6374d725-5459-42da-a6df-849e7df694b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:09.056946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6374d725-5459-42da-a6df-849e7df694b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:09.057615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6374d725-5459-42da-a6df-849e7df694b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:09.058080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6374d725-5459-42da-a6df-849e7df694b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:09.076434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6374d725-5459-42da-a6df-849e7df694b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:09.108014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6374d725-5459-42da-a6df-849e7df694b6 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:09.131802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 824/1656] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:09.814806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04c3bb9-3726-4b5b-813f-6227430f6e93 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:09.816572 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e04c3bb9-3726-4b5b-813f-6227430f6e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:09.836810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04c3bb9-3726-4b5b-813f-6227430f6e93 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:09.851194 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:09.851893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:09.852157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:09.852538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] PATH_INFO: `/v3/projects/a9406983aa784d9ca3a16ddc6bffeb92` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:09.853800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9406983aa784d9ca3a16ddc6bffeb92)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:09.856026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:09.993109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9406983aa784d9ca3a16ddc6bffeb92', 'request_id': 'req-a3d8f14b-668f-4695-8b49-a3d18afb0f44'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:10.005562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9406983aa784d9ca3a16ddc6bffeb92', 'request_id': 'req-a3d8f14b-668f-4695-8b49-a3d18afb0f44'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:10.023121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9406983aa784d9ca3a16ddc6bffeb92', 'request_id': 'req-a3d8f14b-668f-4695-8b49-a3d18afb0f44'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:10.049731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] The token cache is being invalidate because project a9406983aa784d9ca3a16ddc6bffeb92 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:10.050049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:10.052424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a3d8f14b-668f-4695-8b49-a3d18afb0f44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:10.059346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 833/1657] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:09 2022] DELETE /identity/v3/projects/a9406983aa784d9ca3a16ddc6bffeb92 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:10.060053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6374d725-5459-42da-a6df-849e7df694b6 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:10.071986 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6374d725-5459-42da-a6df-849e7df694b6 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:10.343164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:10.343950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:10.344494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:10.344728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] PATH_INFO: `/v3/users/a15972d85de84e369221497aa1715c80` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:10.346402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a15972d85de84e369221497aa1715c80)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:10.349838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:10.467350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a15972d85de84e369221497aa1715c80'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:10.485875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a15972d85de84e369221497aa1715c80'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:10.486709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a15972d85de84e369221497aa1715c80'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:10.487632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a15972d85de84e369221497aa1715c80'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:10.505546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a15972d85de84e369221497aa1715c80'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:10.538322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 825/1658] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:10 2022] DELETE /identity/v3/users/a15972d85de84e369221497aa1715c80 => generated 0 bytes in 478 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:11.099408 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5096e358-c714-4404-9d5d-b991d329bb64 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:11.099972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5096e358-c714-4404-9d5d-b991d329bb64 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:11.099972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5096e358-c714-4404-9d5d-b991d329bb64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:11.196518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-5096e358-c714-4404-9d5d-b991d329bb64 None None] MFA Rules not processed for user `e99e0aa6af854e8aaa7c263c42814e77`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:11.199403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5096e358-c714-4404-9d5d-b991d329bb64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:11.259151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:11.381864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:11.383361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:11.389513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:11.629841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 834/1659] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:11 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 527 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:11.667324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b886b8-2d5b-4594-8787-12b0b3ef6809 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:11.691810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:11.692500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:11.692779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:11.693225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] PATH_INFO: `/v3/projects/670e7230a6e94ef99dfd035d3b37676b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:11.701312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=670e7230a6e94ef99dfd035d3b37676b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:11.702379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:36:11.702379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 26 01:36:11.702379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: Sep 26 01:36:11.703607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:11.905739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '670e7230a6e94ef99dfd035d3b37676b', 'request_id': 'req-46c130a4-23e6-4315-93bd-a8f1db2208a1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:11.953458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '670e7230a6e94ef99dfd035d3b37676b', 'request_id': 'req-46c130a4-23e6-4315-93bd-a8f1db2208a1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:11.967780 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '670e7230a6e94ef99dfd035d3b37676b', 'request_id': 'req-46c130a4-23e6-4315-93bd-a8f1db2208a1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:11.992292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] The token cache is being invalidate because project 670e7230a6e94ef99dfd035d3b37676b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:11.992621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:11.993343 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:11.999289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5096e358-c714-4404-9d5d-b991d329bb64 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:11.999842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 826/1660] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:11 2022] DELETE /identity/v3/projects/670e7230a6e94ef99dfd035d3b37676b => generated 0 bytes in 614 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:12.018619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5096e358-c714-4404-9d5d-b991d329bb64 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:12.142130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:12.142934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:12.143040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:12.143310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] PATH_INFO: `/v3/users/e0b83ab36c574e93bfb6ea01a5e0d667` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:12.146677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0b83ab36c574e93bfb6ea01a5e0d667)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:12.147741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: WARNING py.warnings [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 26 01:36:12.147741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: warnings.warn(msg) Sep 26 01:36:12.147741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: Sep 26 01:36:12.149012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:12.322886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0b83ab36c574e93bfb6ea01a5e0d667'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:12.350953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0b83ab36c574e93bfb6ea01a5e0d667'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:12.352999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0b83ab36c574e93bfb6ea01a5e0d667'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:12.355635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0b83ab36c574e93bfb6ea01a5e0d667'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:12.418131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0b83ab36c574e93bfb6ea01a5e0d667'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:12.494869 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 835/1661] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:11 2022] DELETE /identity/v3/users/e0b83ab36c574e93bfb6ea01a5e0d667 => generated 0 bytes in 492 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:13.176131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:13.176131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:13.201360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c130a4-23e6-4315-93bd-a8f1db2208a1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:13.334526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:13.335184 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:13.335471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:13.335731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] PATH_INFO: `/v3/users/b6f18267454a4d148dd701ecb1109aac` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:13.336920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6f18267454a4d148dd701ecb1109aac)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:13.345163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:13.402894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:13.406680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:13.426316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:13.448009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6f18267454a4d148dd701ecb1109aac'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:13.449669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7e541f-903a-426d-96fe-d5a320eca0e0 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:13.463263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:13.465839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6f18267454a4d148dd701ecb1109aac'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:13.467032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6f18267454a4d148dd701ecb1109aac'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:13.467384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:13.467694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6f18267454a4d148dd701ecb1109aac'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:13.467823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:13.468105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] PATH_INFO: `/v3/projects/964d347c1bf8455d81a115b3c805e622` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:13.471571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=964d347c1bf8455d81a115b3c805e622)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:13.476033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:13.483706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-58c7448d-92f7-4086-a1e8-2c9cfa1a4355 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6f18267454a4d148dd701ecb1109aac'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:13.516233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 827/1662] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:13 2022] DELETE /identity/v3/users/b6f18267454a4d148dd701ecb1109aac => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:13.520399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0474ee98-9bae-4a31-ac05-d8a892bbdad7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:13.520920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0474ee98-9bae-4a31-ac05-d8a892bbdad7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:13.521409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0474ee98-9bae-4a31-ac05-d8a892bbdad7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:13.545933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:13.561985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0474ee98-9bae-4a31-ac05-d8a892bbdad7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:13.565856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0474ee98-9bae-4a31-ac05-d8a892bbdad7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:13.604316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '964d347c1bf8455d81a115b3c805e622', 'request_id': 'req-9fd90846-342e-47f1-bf09-151a7b819b97'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:13.617286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '964d347c1bf8455d81a115b3c805e622', 'request_id': 'req-9fd90846-342e-47f1-bf09-151a7b819b97'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:13.630790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '964d347c1bf8455d81a115b3c805e622', 'request_id': 'req-9fd90846-342e-47f1-bf09-151a7b819b97'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:13.672546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] The token cache is being invalidate because project 964d347c1bf8455d81a115b3c805e622 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:13.672849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:13.674759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9fd90846-342e-47f1-bf09-151a7b819b97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:13.685426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 836/1663] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:13 2022] DELETE /identity/v3/projects/964d347c1bf8455d81a115b3c805e622 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:13.690828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbd262-ade8-455c-8672-12d0f06e85cf None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:13.694737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbd262-ade8-455c-8672-12d0f06e85cf None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:13.695299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbd262-ade8-455c-8672-12d0f06e85cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:13.720043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-ddfbd262-ade8-455c-8672-12d0f06e85cf None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:13.723202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ddfbd262-ade8-455c-8672-12d0f06e85cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:13.751220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 828/1664] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:13.805725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0474ee98-9bae-4a31-ac05-d8a892bbdad7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:13.807614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0474ee98-9bae-4a31-ac05-d8a892bbdad7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:13.875269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 837/1665] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:13.884473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda0b584-a13f-4f77-90d4-8be0ec559669 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:13.884935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda0b584-a13f-4f77-90d4-8be0ec559669 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:13.885224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda0b584-a13f-4f77-90d4-8be0ec559669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:13.937950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-dda0b584-a13f-4f77-90d4-8be0ec559669 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:13.939185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-dda0b584-a13f-4f77-90d4-8be0ec559669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:13.945796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0474ee98-9bae-4a31-ac05-d8a892bbdad7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:13.952871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab0aab5-49d0-4951-b886-95ce29ed1b38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:13.954005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab0aab5-49d0-4951-b886-95ce29ed1b38 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:13.956059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab0aab5-49d0-4951-b886-95ce29ed1b38 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:13.957029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab0aab5-49d0-4951-b886-95ce29ed1b38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:13.969712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 838/1666] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:13.975110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda0b584-a13f-4f77-90d4-8be0ec559669 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:13.992233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab0aab5-49d0-4951-b886-95ce29ed1b38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:13.994524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab0aab5-49d0-4951-b886-95ce29ed1b38 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:14.007069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda0b584-a13f-4f77-90d4-8be0ec559669 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:14.012186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e72971-f197-487e-84e5-0aeb5b1e3097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:14.012847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e72971-f197-487e-84e5-0aeb5b1e3097 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:14.013125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e72971-f197-487e-84e5-0aeb5b1e3097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:14.013401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e72971-f197-487e-84e5-0aeb5b1e3097 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:14.014063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e72971-f197-487e-84e5-0aeb5b1e3097 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:14.014371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e72971-f197-487e-84e5-0aeb5b1e3097 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:14.016001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e72971-f197-487e-84e5-0aeb5b1e3097 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:14.022939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 829/1667] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:36:14.049779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 839/1668] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:14.054329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab0aab5-49d0-4951-b886-95ce29ed1b38 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:14.078087 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab0aab5-49d0-4951-b886-95ce29ed1b38 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:14.090450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0cb82f-e928-456b-acbd-74d5f0ac728c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:14.091169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0cb82f-e928-456b-acbd-74d5f0ac728c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:14.092636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0cb82f-e928-456b-acbd-74d5f0ac728c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:14.092636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0cb82f-e928-456b-acbd-74d5f0ac728c admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:14.093396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0cb82f-e928-456b-acbd-74d5f0ac728c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:14.098275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0cb82f-e928-456b-acbd-74d5f0ac728c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:14.170010 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 830/1669] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:14 2022] POST /identity/v3/projects => generated 369 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:14.174162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e72971-f197-487e-84e5-0aeb5b1e3097 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:14.186869 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e72971-f197-487e-84e5-0aeb5b1e3097 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:14.317239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1846319b-3dfc-4337-b6e7-f0bf9c010b17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:14.317879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1846319b-3dfc-4337-b6e7-f0bf9c010b17 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:14.318166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1846319b-3dfc-4337-b6e7-f0bf9c010b17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:14.318448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1846319b-3dfc-4337-b6e7-f0bf9c010b17 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:14.319110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1846319b-3dfc-4337-b6e7-f0bf9c010b17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:14.325713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1846319b-3dfc-4337-b6e7-f0bf9c010b17 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:14.393952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 840/1670] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:14 2022] POST /identity/v3/users => generated 285 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:14.400633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0cb82f-e928-456b-acbd-74d5f0ac728c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:14.414947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0cb82f-e928-456b-acbd-74d5f0ac728c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:14.421221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edd06b3-668a-4a8e-9725-a088cfcddbbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:14.421430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edd06b3-668a-4a8e-9725-a088cfcddbbc admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:14.421678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edd06b3-668a-4a8e-9725-a088cfcddbbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:14.421952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edd06b3-668a-4a8e-9725-a088cfcddbbc admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:14.423318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edd06b3-668a-4a8e-9725-a088cfcddbbc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:14.425168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edd06b3-668a-4a8e-9725-a088cfcddbbc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:14.453254 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 831/1671] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:14 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:14.457673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1846319b-3dfc-4337-b6e7-f0bf9c010b17 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:14.476586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1846319b-3dfc-4337-b6e7-f0bf9c010b17 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:14.488478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031bb7c2-9ac1-4646-889a-1e2ed92958cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:14.489441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031bb7c2-9ac1-4646-889a-1e2ed92958cd admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:14.490089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031bb7c2-9ac1-4646-889a-1e2ed92958cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:14.501708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031bb7c2-9ac1-4646-889a-1e2ed92958cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:14.502566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031bb7c2-9ac1-4646-889a-1e2ed92958cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:14.504368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031bb7c2-9ac1-4646-889a-1e2ed92958cd admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:14.527413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 841/1672] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:14 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:14.529557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edd06b3-668a-4a8e-9725-a088cfcddbbc admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:14.543967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edd06b3-668a-4a8e-9725-a088cfcddbbc admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:14.552717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35c7d02-ddcc-4dd0-a8c3-ba453fad8fd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:14.553901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35c7d02-ddcc-4dd0-a8c3-ba453fad8fd5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:14.554480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35c7d02-ddcc-4dd0-a8c3-ba453fad8fd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:14.555006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35c7d02-ddcc-4dd0-a8c3-ba453fad8fd5 admin admin] PATH_INFO: `/v3/projects/513260c0f2484a2d885d0e0a66f4e174/users/5184f8f717d447cfbb7bab80d4cf000b/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:14.560577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031bb7c2-9ac1-4646-889a-1e2ed92958cd admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:14.565635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-031bb7c2-9ac1-4646-889a-1e2ed92958cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:14.594213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031bb7c2-9ac1-4646-889a-1e2ed92958cd admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:14.620048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:14.621504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:14.622561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:14.623522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35c7d02-ddcc-4dd0-a8c3-ba453fad8fd5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=513260c0f2484a2d885d0e0a66f4e174, user_id=5184f8f717d447cfbb7bab80d4cf000b, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:14.623939 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] PATH_INFO: `/v3/projects/c06fcc7562514314971e32566197db67` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:14.627372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35c7d02-ddcc-4dd0-a8c3-ba453fad8fd5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:14.628324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c06fcc7562514314971e32566197db67)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:14.630680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:14.640684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:14.653801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:14.667968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 832/1673] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:14 2022] PUT /identity/v3/projects/513260c0f2484a2d885d0e0a66f4e174/users/5184f8f717d447cfbb7bab80d4cf000b/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:14.675270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa6d1c7-6344-48e8-9e4b-a23295873854 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:14.685130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa6d1c7-6344-48e8-9e4b-a23295873854 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:14.685605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa6d1c7-6344-48e8-9e4b-a23295873854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:14.747194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6fa6d1c7-6344-48e8-9e4b-a23295873854 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:14.750662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6fa6d1c7-6344-48e8-9e4b-a23295873854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:14.782144 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c06fcc7562514314971e32566197db67', 'request_id': 'req-db721a9b-c1fb-4053-813d-72579930d8e8'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:14.794007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c06fcc7562514314971e32566197db67', 'request_id': 'req-db721a9b-c1fb-4053-813d-72579930d8e8'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:14.809556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c06fcc7562514314971e32566197db67', 'request_id': 'req-db721a9b-c1fb-4053-813d-72579930d8e8'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:14.832544 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] The token cache is being invalidate because project c06fcc7562514314971e32566197db67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:14.833123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:14.834137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db721a9b-c1fb-4053-813d-72579930d8e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:14.836187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 842/1674] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:14 2022] DELETE /identity/v3/projects/c06fcc7562514314971e32566197db67 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:14.841426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277d698a-d013-42cd-8097-0742aa682f37 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:14.841929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277d698a-d013-42cd-8097-0742aa682f37 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:14.842339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277d698a-d013-42cd-8097-0742aa682f37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:14.885098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-277d698a-d013-42cd-8097-0742aa682f37 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:14.886667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-277d698a-d013-42cd-8097-0742aa682f37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:14.903613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 833/1675] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:14.906229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 843/1676] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:14.917385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da3be3-1bf9-473a-9975-03b1077295a7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:14.917679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da3be3-1bf9-473a-9975-03b1077295a7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:14.918339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da3be3-1bf9-473a-9975-03b1077295a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:14.931931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277d698a-d013-42cd-8097-0742aa682f37 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:14.933801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-277d698a-d013-42cd-8097-0742aa682f37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:14.973365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-00da3be3-1bf9-473a-9975-03b1077295a7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:14.977685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-00da3be3-1bf9-473a-9975-03b1077295a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:15.002650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 834/1677] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:15.007506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00da3be3-1bf9-473a-9975-03b1077295a7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:15.009237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277d698a-d013-42cd-8097-0742aa682f37 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:15.014570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6af4a8-5804-4e4c-86da-227176e92ef7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:15.015756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6af4a8-5804-4e4c-86da-227176e92ef7 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:15.016231 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6af4a8-5804-4e4c-86da-227176e92ef7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:15.016930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6af4a8-5804-4e4c-86da-227176e92ef7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:15.028629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6af4a8-5804-4e4c-86da-227176e92ef7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:15.029255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00da3be3-1bf9-473a-9975-03b1077295a7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:15.031547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6af4a8-5804-4e4c-86da-227176e92ef7 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:15.040029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb9aaa8-bfcf-4877-b96d-0f253224ee8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:15.042791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb9aaa8-bfcf-4877-b96d-0f253224ee8a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:15.043076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb9aaa8-bfcf-4877-b96d-0f253224ee8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:15.043350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb9aaa8-bfcf-4877-b96d-0f253224ee8a admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:15.044670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb9aaa8-bfcf-4877-b96d-0f253224ee8a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:15.044989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb9aaa8-bfcf-4877-b96d-0f253224ee8a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:15.046615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb9aaa8-bfcf-4877-b96d-0f253224ee8a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:15.047474 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 844/1678] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:36:15.055412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fc1db2-f45e-410f-853b-762757605803 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:15.056288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fc1db2-f45e-410f-853b-762757605803 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:15.056680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fc1db2-f45e-410f-853b-762757605803 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:15.060728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 845/1679] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:36:15 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:36:15.062754 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 835/1680] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:15.067077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fc1db2-f45e-410f-853b-762757605803 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:15.088376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fc1db2-f45e-410f-853b-762757605803 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:15.093125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90be916e-0f94-4944-a9da-198c64173677 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:15.093825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90be916e-0f94-4944-a9da-198c64173677 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:15.094181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90be916e-0f94-4944-a9da-198c64173677 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:15.094527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90be916e-0f94-4944-a9da-198c64173677 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:15.095242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90be916e-0f94-4944-a9da-198c64173677 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:15.099369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90be916e-0f94-4944-a9da-198c64173677 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:15.142087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 846/1681] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:15 2022] POST /identity/v3/projects => generated 385 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:15.146417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb9aaa8-bfcf-4877-b96d-0f253224ee8a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:15.160420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb9aaa8-bfcf-4877-b96d-0f253224ee8a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:15.312290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba08609-3dd2-4230-bcfb-af141673ed3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:15.312964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba08609-3dd2-4230-bcfb-af141673ed3a admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:15.313246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba08609-3dd2-4230-bcfb-af141673ed3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:15.313524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba08609-3dd2-4230-bcfb-af141673ed3a admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:15.314298 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba08609-3dd2-4230-bcfb-af141673ed3a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:15.316084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba08609-3dd2-4230-bcfb-af141673ed3a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:15.364010 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 836/1682] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:15 2022] POST /identity/v3/users => generated 293 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:15.368333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90be916e-0f94-4944-a9da-198c64173677 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:15.385531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90be916e-0f94-4944-a9da-198c64173677 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:15.403160 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d339a839-6c9f-49c6-8235-baf06bcb2382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:15.403957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d339a839-6c9f-49c6-8235-baf06bcb2382 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:15.404320 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d339a839-6c9f-49c6-8235-baf06bcb2382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:15.404698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d339a839-6c9f-49c6-8235-baf06bcb2382 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:15.405503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d339a839-6c9f-49c6-8235-baf06bcb2382 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:15.411828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d339a839-6c9f-49c6-8235-baf06bcb2382 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:15.440747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 847/1683] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:15 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:15.443746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba08609-3dd2-4230-bcfb-af141673ed3a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:15.460449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba08609-3dd2-4230-bcfb-af141673ed3a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:15.472622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c92cee3-43d6-4468-b0bf-8a2d3f8ce842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:15.475166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c92cee3-43d6-4468-b0bf-8a2d3f8ce842 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:15.475456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c92cee3-43d6-4468-b0bf-8a2d3f8ce842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:15.476132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c92cee3-43d6-4468-b0bf-8a2d3f8ce842 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:15.476639 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c92cee3-43d6-4468-b0bf-8a2d3f8ce842 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:15.479296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c92cee3-43d6-4468-b0bf-8a2d3f8ce842 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:15.508143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 837/1684] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:15 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:15.513057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d339a839-6c9f-49c6-8235-baf06bcb2382 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:15.525165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d339a839-6c9f-49c6-8235-baf06bcb2382 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:15.531029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999ba613-8f46-4329-ad49-75bfc33e9c75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:15.532588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ba613-8f46-4329-ad49-75bfc33e9c75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:15.533061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ba613-8f46-4329-ad49-75bfc33e9c75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:15.533573 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999ba613-8f46-4329-ad49-75bfc33e9c75 admin admin] PATH_INFO: `/v3/projects/e95f9dadfbb744668231b3ada7960483/users/d0a84852d7834b61808c9f6d291353a1/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:15.555169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999ba613-8f46-4329-ad49-75bfc33e9c75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e95f9dadfbb744668231b3ada7960483, user_id=d0a84852d7834b61808c9f6d291353a1, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:15.559454 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999ba613-8f46-4329-ad49-75bfc33e9c75 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:15.564243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0254e-ef63-47e4-abd6-e39b0067ad3b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:15.568735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0254e-ef63-47e4-abd6-e39b0067ad3b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:15.569273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e0254e-ef63-47e4-abd6-e39b0067ad3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:15.606459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 848/1685] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:15 2022] PUT /identity/v3/projects/e95f9dadfbb744668231b3ada7960483/users/d0a84852d7834b61808c9f6d291353a1/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:15.621688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82af18c6-6595-4df1-9f48-c7b476d5247f None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:15.621944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82af18c6-6595-4df1-9f48-c7b476d5247f None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:15.622281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82af18c6-6595-4df1-9f48-c7b476d5247f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:15.626183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c5e0254e-ef63-47e4-abd6-e39b0067ad3b None None] MFA Rules not processed for user `427016f8da514dc4bbed21de08207dbb`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:15.627431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c5e0254e-ef63-47e4-abd6-e39b0067ad3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:15.704175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-82af18c6-6595-4df1-9f48-c7b476d5247f None None] MFA Rules not processed for user `d0a84852d7834b61808c9f6d291353a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:15.708590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-82af18c6-6595-4df1-9f48-c7b476d5247f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:15.736365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:15.767885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:15.774261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 838/1686] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:36:15 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:15.921904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 849/1687] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:15 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:15.926750 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e0254e-ef63-47e4-abd6-e39b0067ad3b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:15.969843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82af18c6-6595-4df1-9f48-c7b476d5247f None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:15.971290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-82af18c6-6595-4df1-9f48-c7b476d5247f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:16.001806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82af18c6-6595-4df1-9f48-c7b476d5247f None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:16.014324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e72c78a-2f84-4109-8943-24996d47ee60 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:16.014974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e72c78a-2f84-4109-8943-24996d47ee60 service placement] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.015254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e72c78a-2f84-4109-8943-24996d47ee60 service placement] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.015529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e72c78a-2f84-4109-8943-24996d47ee60 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.049524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e72c78a-2f84-4109-8943-24996d47ee60 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:16.051186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e72c78a-2f84-4109-8943-24996d47ee60 service placement] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:16.080274 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 850/1688] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:15 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:36:16.087223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e72c78a-2f84-4109-8943-24996d47ee60 service placement] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:16.094496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0e72c78a-2f84-4109-8943-24996d47ee60 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:16.095318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e0254e-ef63-47e4-abd6-e39b0067ad3b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:16.102474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9564ac-b228-484e-81c9-fc4e9edc25e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:16.103486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9564ac-b228-484e-81c9-fc4e9edc25e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.103997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9564ac-b228-484e-81c9-fc4e9edc25e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.104509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9564ac-b228-484e-81c9-fc4e9edc25e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.105363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9564ac-b228-484e-81c9-fc4e9edc25e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:16.107536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9564ac-b228-484e-81c9-fc4e9edc25e3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:16.168878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e72c78a-2f84-4109-8943-24996d47ee60 service placement] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:16.179154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 839/1689] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:15 2022] POST /identity/v3/projects => generated 385 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:16.180133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9564ac-b228-484e-81c9-fc4e9edc25e3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:16.183427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:16.184852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.185150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.185447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] PATH_INFO: `/v3/projects/f7cff87a197d4ddfbaeb0ac3d6d87fa9` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.191190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7cff87a197d4ddfbaeb0ac3d6d87fa9)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:16.196766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:16.292947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9564ac-b228-484e-81c9-fc4e9edc25e3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:16.301260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ec678a-bf7a-4888-b9d3-3bcf2e3b85bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:16.303422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ec678a-bf7a-4888-b9d3-3bcf2e3b85bd admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.307043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ec678a-bf7a-4888-b9d3-3bcf2e3b85bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.308275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ec678a-bf7a-4888-b9d3-3bcf2e3b85bd admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.308731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ec678a-bf7a-4888-b9d3-3bcf2e3b85bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:16.311448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ec678a-bf7a-4888-b9d3-3bcf2e3b85bd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:16.359117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 840/1690] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:16 2022] POST /identity/v3/users => generated 299 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:16.360778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ec678a-bf7a-4888-b9d3-3bcf2e3b85bd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:16.360886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7cff87a197d4ddfbaeb0ac3d6d87fa9', 'request_id': 'req-7fcf086f-3118-4d67-a5ad-ab136bab2059'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:16.373421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ec678a-bf7a-4888-b9d3-3bcf2e3b85bd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:16.380371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7cff87a197d4ddfbaeb0ac3d6d87fa9', 'request_id': 'req-7fcf086f-3118-4d67-a5ad-ab136bab2059'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:16.404798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7cff87a197d4ddfbaeb0ac3d6d87fa9', 'request_id': 'req-7fcf086f-3118-4d67-a5ad-ab136bab2059'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:16.441900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] The token cache is being invalidate because project f7cff87a197d4ddfbaeb0ac3d6d87fa9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:16.442224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:16.442945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7fcf086f-3118-4d67-a5ad-ab136bab2059 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:16.449359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 851/1691] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:16 2022] DELETE /identity/v3/projects/f7cff87a197d4ddfbaeb0ac3d6d87fa9 => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:16.451455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad82cd-1b5c-4e6b-ba90-43993c65f981 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.451747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad82cd-1b5c-4e6b-ba90-43993c65f981 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.452030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad82cd-1b5c-4e6b-ba90-43993c65f981 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.490903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a384563-1318-461a-8413-9597057e70d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:16.491600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a384563-1318-461a-8413-9597057e70d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.491825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a384563-1318-461a-8413-9597057e70d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.492108 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a384563-1318-461a-8413-9597057e70d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.492819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a384563-1318-461a-8413-9597057e70d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:16.495006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a384563-1318-461a-8413-9597057e70d1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:16.512000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-47ad82cd-1b5c-4e6b-ba90-43993c65f981 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:16.513332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-47ad82cd-1b5c-4e6b-ba90-43993c65f981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:16.518667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 841/1692] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:16 2022] GET /identity/v3/roles => generated 1420 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:16.523875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a384563-1318-461a-8413-9597057e70d1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:16.526275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5a384563-1318-461a-8413-9597057e70d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:16.537621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:16.544897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a384563-1318-461a-8413-9597057e70d1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:16.546411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 852/1693] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:16.551479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eb1b69-dd0f-4a82-9af7-e0df518a91ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:16.552354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eb1b69-dd0f-4a82-9af7-e0df518a91ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.553234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eb1b69-dd0f-4a82-9af7-e0df518a91ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.553516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eb1b69-dd0f-4a82-9af7-e0df518a91ab admin admin] PATH_INFO: `/v3/projects/2135e08f67b54af6ba37caf73a0d2629/users/d149dbf5ef87428ba6009e1d16a0e6ae/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.554732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac996fee-9758-4735-820d-352236cc392c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.555034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac996fee-9758-4735-820d-352236cc392c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.555321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac996fee-9758-4735-820d-352236cc392c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.578249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06eb1b69-dd0f-4a82-9af7-e0df518a91ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=2135e08f67b54af6ba37caf73a0d2629, user_id=d149dbf5ef87428ba6009e1d16a0e6ae, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:16.592531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06eb1b69-dd0f-4a82-9af7-e0df518a91ab admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:16.601991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-ac996fee-9758-4735-820d-352236cc392c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:16.603887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ac996fee-9758-4735-820d-352236cc392c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:16.642935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 842/1694] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:16 2022] PUT /identity/v3/projects/2135e08f67b54af6ba37caf73a0d2629/users/d149dbf5ef87428ba6009e1d16a0e6ae/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:16.649750 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eb1b69-dd0f-4a82-9af7-e0df518a91ab admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:16.740004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 853/1695] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:16.744825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac996fee-9758-4735-820d-352236cc392c None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:16.762540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac996fee-9758-4735-820d-352236cc392c None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:16.776389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b6b8d-2f03-44bf-8290-4bc924642706 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:16.777071 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b6b8d-2f03-44bf-8290-4bc924642706 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.777315 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b6b8d-2f03-44bf-8290-4bc924642706 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.777585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b6b8d-2f03-44bf-8290-4bc924642706 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.778523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69b6b8d-2f03-44bf-8290-4bc924642706 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:16.778523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69b6b8d-2f03-44bf-8290-4bc924642706 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:16.780174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69b6b8d-2f03-44bf-8290-4bc924642706 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:16.810489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 854/1696] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:36:16.816224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b6b8d-2f03-44bf-8290-4bc924642706 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:16.829497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b6b8d-2f03-44bf-8290-4bc924642706 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:16.844939 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50127c84-b05f-4de1-8140-f0142d28e116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:16.845566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50127c84-b05f-4de1-8140-f0142d28e116 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.845848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50127c84-b05f-4de1-8140-f0142d28e116 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.846127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50127c84-b05f-4de1-8140-f0142d28e116 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.846862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50127c84-b05f-4de1-8140-f0142d28e116 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:16.849239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eb1b69-dd0f-4a82-9af7-e0df518a91ab admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:16.850870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50127c84-b05f-4de1-8140-f0142d28e116 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:16.862276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b2df05-f1ea-4940-97c9-7381a54cc559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:16.863155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b2df05-f1ea-4940-97c9-7381a54cc559 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:16.863155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b2df05-f1ea-4940-97c9-7381a54cc559 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:16.863412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b2df05-f1ea-4940-97c9-7381a54cc559 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:16.864481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b2df05-f1ea-4940-97c9-7381a54cc559 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:16.868278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b2df05-f1ea-4940-97c9-7381a54cc559 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:16.876367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:16.891192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 843/1697] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:16 2022] GET /identity/v3/roles => generated 1420 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:16.893384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b2df05-f1ea-4940-97c9-7381a54cc559 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:16.908542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 855/1698] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:16 2022] POST /identity/v3/projects => generated 373 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:36:16.912775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50127c84-b05f-4de1-8140-f0142d28e116 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:16.915197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b2df05-f1ea-4940-97c9-7381a54cc559 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:17.038059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50127c84-b05f-4de1-8140-f0142d28e116 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:17.045910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f38928d-0a6d-4aa1-b8fd-c9d662b5553c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:17.046639 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f38928d-0a6d-4aa1-b8fd-c9d662b5553c admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:17.046925 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f38928d-0a6d-4aa1-b8fd-c9d662b5553c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:17.047207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f38928d-0a6d-4aa1-b8fd-c9d662b5553c admin admin] PATH_INFO: `/v3/projects/2135e08f67b54af6ba37caf73a0d2629/users/d149dbf5ef87428ba6009e1d16a0e6ae/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:17.050910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f38928d-0a6d-4aa1-b8fd-c9d662b5553c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2135e08f67b54af6ba37caf73a0d2629, user_id=d149dbf5ef87428ba6009e1d16a0e6ae, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:17.055525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f38928d-0a6d-4aa1-b8fd-c9d662b5553c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:17.060647 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594cc795-9721-4767-8eb3-30313fc026c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:17.061340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594cc795-9721-4767-8eb3-30313fc026c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:17.061631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594cc795-9721-4767-8eb3-30313fc026c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:17.061893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594cc795-9721-4767-8eb3-30313fc026c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:17.062775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594cc795-9721-4767-8eb3-30313fc026c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:17.064809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594cc795-9721-4767-8eb3-30313fc026c0 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:17.095859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 844/1699] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:16 2022] PUT /identity/v3/projects/2135e08f67b54af6ba37caf73a0d2629/users/d149dbf5ef87428ba6009e1d16a0e6ae/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:17.107344 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca9c51b-e83b-4a9a-985d-d65c1280b576 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:17.107900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca9c51b-e83b-4a9a-985d-d65c1280b576 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:17.108247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca9c51b-e83b-4a9a-985d-d65c1280b576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:17.113710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 856/1700] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:16 2022] POST /identity/v3/users => generated 287 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:36:17.117466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594cc795-9721-4767-8eb3-30313fc026c0 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:17.134930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594cc795-9721-4767-8eb3-30313fc026c0 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:17.166004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9ca9c51b-e83b-4a9a-985d-d65c1280b576 None None] MFA Rules not processed for user `d149dbf5ef87428ba6009e1d16a0e6ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:17.172639 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9ca9c51b-e83b-4a9a-985d-d65c1280b576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:17.252280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f20846-c763-4bbb-8ba0-76e51034cb9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:17.252958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f20846-c763-4bbb-8ba0-76e51034cb9a admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:17.253367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f20846-c763-4bbb-8ba0-76e51034cb9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:17.253490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f20846-c763-4bbb-8ba0-76e51034cb9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:17.254164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f20846-c763-4bbb-8ba0-76e51034cb9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:17.255798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f20846-c763-4bbb-8ba0-76e51034cb9a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:17.283203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 857/1701] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:17 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:36:17.286680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f20846-c763-4bbb-8ba0-76e51034cb9a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:17.306338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f20846-c763-4bbb-8ba0-76e51034cb9a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:17.334819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8544585-c71f-46d5-9a00-be4f16bbf9da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:17.335452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8544585-c71f-46d5-9a00-be4f16bbf9da admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:17.335562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:17.335733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8544585-c71f-46d5-9a00-be4f16bbf9da admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:17.336003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8544585-c71f-46d5-9a00-be4f16bbf9da admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:17.336720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8544585-c71f-46d5-9a00-be4f16bbf9da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:17.339928 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8544585-c71f-46d5-9a00-be4f16bbf9da admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:17.361790 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 845/1702] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:17 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:17.365395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 858/1703] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:17 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:17.370372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8544585-c71f-46d5-9a00-be4f16bbf9da admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:17.370653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca9c51b-e83b-4a9a-985d-d65c1280b576 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:17.372398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b8544585-c71f-46d5-9a00-be4f16bbf9da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:17.427761 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca9c51b-e83b-4a9a-985d-d65c1280b576 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:17.437145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2cbe10-9a01-4a28-868f-3d8259be193a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:17.438292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2cbe10-9a01-4a28-868f-3d8259be193a admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:17.438781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2cbe10-9a01-4a28-868f-3d8259be193a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:17.439208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2cbe10-9a01-4a28-868f-3d8259be193a admin admin] PATH_INFO: `/v3/projects/b4e2ab570b2c461f9ad8fc19dcbe7f0f/users/13b7ef0b77114af6b5d1d162a5e9434f/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:17.467384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2cbe10-9a01-4a28-868f-3d8259be193a admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4e2ab570b2c461f9ad8fc19dcbe7f0f, user_id=13b7ef0b77114af6b5d1d162a5e9434f, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:17.471640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2cbe10-9a01-4a28-868f-3d8259be193a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:17.514985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 846/1704] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:17 2022] PUT /identity/v3/projects/b4e2ab570b2c461f9ad8fc19dcbe7f0f/users/13b7ef0b77114af6b5d1d162a5e9434f/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:17.515610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8544585-c71f-46d5-9a00-be4f16bbf9da admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:17.517480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f42135b-38c0-49c7-a6f0-e2b03d31df2a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:17.518170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f42135b-38c0-49c7-a6f0-e2b03d31df2a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:17.518426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f42135b-38c0-49c7-a6f0-e2b03d31df2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:17.526013 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:17.598364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d666e25-e6a8-4c2b-bceb-d7691d404318 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:17.599113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d666e25-e6a8-4c2b-bceb-d7691d404318 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:17.599525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d666e25-e6a8-4c2b-bceb-d7691d404318 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:17.600563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d666e25-e6a8-4c2b-bceb-d7691d404318 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:17.600682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6f42135b-38c0-49c7-a6f0-e2b03d31df2a None None] MFA Rules not processed for user `13b7ef0b77114af6b5d1d162a5e9434f`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:17.601453 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6f42135b-38c0-49c7-a6f0-e2b03d31df2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:17.737782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d666e25-e6a8-4c2b-bceb-d7691d404318 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:17.743480 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d666e25-e6a8-4c2b-bceb-d7691d404318 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:17.761223 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 847/1705] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:17.762477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f42135b-38c0-49c7-a6f0-e2b03d31df2a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:17.777752 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 859/1706] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:17 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:17.874447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d666e25-e6a8-4c2b-bceb-d7691d404318 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:17.949668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f42135b-38c0-49c7-a6f0-e2b03d31df2a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:17.954825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0c1f24-e7bc-4274-ab40-19e0a1a1775c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:17.955658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0c1f24-e7bc-4274-ab40-19e0a1a1775c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:17.956586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0c1f24-e7bc-4274-ab40-19e0a1a1775c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:17.957005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0c1f24-e7bc-4274-ab40-19e0a1a1775c admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:17.958069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0c1f24-e7bc-4274-ab40-19e0a1a1775c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:17.960216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0c1f24-e7bc-4274-ab40-19e0a1a1775c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:18.032527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 848/1707] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:17 2022] POST /identity/v3/projects => generated 371 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:36:18.036403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0c1f24-e7bc-4274-ab40-19e0a1a1775c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:18.046409 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d666e25-e6a8-4c2b-bceb-d7691d404318 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:18.112909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:18.150105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0c1f24-e7bc-4274-ab40-19e0a1a1775c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:18.155511 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:18.156239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:18.156616 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:18.156925 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] PATH_INFO: `/v3/users/d0a84852d7834b61808c9f6d291353a1` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:18.158242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] RBAC: Authorizing `identity:delete_user(user_id=d0a84852d7834b61808c9f6d291353a1)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:18.159245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd467070-a1c2-43ff-a64b-0672e4a92e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:18.159874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd467070-a1c2-43ff-a64b-0672e4a92e49 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:18.160157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd467070-a1c2-43ff-a64b-0672e4a92e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:18.160444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:18.160549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd467070-a1c2-43ff-a64b-0672e4a92e49 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:18.161092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd467070-a1c2-43ff-a64b-0672e4a92e49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:18.163039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd467070-a1c2-43ff-a64b-0672e4a92e49 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:18.188696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:18.241424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd0a84852d7834b61808c9f6d291353a1'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:18.250495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 849/1708] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:18 2022] POST /identity/v3/users => generated 292 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:18.257770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd467070-a1c2-43ff-a64b-0672e4a92e49 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:18.267235 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd0a84852d7834b61808c9f6d291353a1'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:18.267976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd0a84852d7834b61808c9f6d291353a1'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:18.270389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd0a84852d7834b61808c9f6d291353a1'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:18.274273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd467070-a1c2-43ff-a64b-0672e4a92e49 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:18.281089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76843bae-071c-4a81-94e8-60173a4affab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:18.282115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76843bae-071c-4a81-94e8-60173a4affab admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:18.283275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76843bae-071c-4a81-94e8-60173a4affab admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:18.283758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76843bae-071c-4a81-94e8-60173a4affab admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:18.285596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76843bae-071c-4a81-94e8-60173a4affab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:18.287484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76843bae-071c-4a81-94e8-60173a4affab admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:18.301301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 850/1709] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:18 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:18.305853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76843bae-071c-4a81-94e8-60173a4affab admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:18.310010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-da6d03a7-9a36-4c15-abd9-8ac364f18edf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd0a84852d7834b61808c9f6d291353a1'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:18.315230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-76843bae-071c-4a81-94e8-60173a4affab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:18.336416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76843bae-071c-4a81-94e8-60173a4affab admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:18.338949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 860/1710] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:17 2022] DELETE /identity/v3/users/d0a84852d7834b61808c9f6d291353a1 => generated 0 bytes in 465 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:18.346388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774aab04-0be4-4736-93a0-2e149b053ffb None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:18.346388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774aab04-0be4-4736-93a0-2e149b053ffb None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:18.346388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774aab04-0be4-4736-93a0-2e149b053ffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:18.397890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-774aab04-0be4-4736-93a0-2e149b053ffb None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:18.401132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-774aab04-0be4-4736-93a0-2e149b053ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:18.459133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927d4b80-f99e-465b-a0a2-1aee9a235703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:18.459992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927d4b80-f99e-465b-a0a2-1aee9a235703 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:18.460366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927d4b80-f99e-465b-a0a2-1aee9a235703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:18.460647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927d4b80-f99e-465b-a0a2-1aee9a235703 admin admin] PATH_INFO: `/v3/projects/a4ec12eace284a28a8cf620f04718707/users/ff49582d0f2747eab23ec79b63640e4b/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:18.489062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927d4b80-f99e-465b-a0a2-1aee9a235703 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4ec12eace284a28a8cf620f04718707, user_id=ff49582d0f2747eab23ec79b63640e4b, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:18.492324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927d4b80-f99e-465b-a0a2-1aee9a235703 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:18.523653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 851/1711] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:18 2022] PUT /identity/v3/projects/a4ec12eace284a28a8cf620f04718707/users/ff49582d0f2747eab23ec79b63640e4b/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:18.528202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927d4b80-f99e-465b-a0a2-1aee9a235703 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:18.562818 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 861/1712] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:18.627952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774aab04-0be4-4736-93a0-2e149b053ffb None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:18.632707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-774aab04-0be4-4736-93a0-2e149b053ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:18.685614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927d4b80-f99e-465b-a0a2-1aee9a235703 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:18.704290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4845a5b-8be3-496b-b3c6-7019390649e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:18.705224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4845a5b-8be3-496b-b3c6-7019390649e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:18.706256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4845a5b-8be3-496b-b3c6-7019390649e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:18.707167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4845a5b-8be3-496b-b3c6-7019390649e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:18.707968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4845a5b-8be3-496b-b3c6-7019390649e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:18.713363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4845a5b-8be3-496b-b3c6-7019390649e1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:18.740527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 852/1713] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:18 2022] GET /identity/v3/roles => generated 1420 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:36:18.744654 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4845a5b-8be3-496b-b3c6-7019390649e1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:18.766770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4845a5b-8be3-496b-b3c6-7019390649e1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:18.772131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774aab04-0be4-4736-93a0-2e149b053ffb None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:18.781623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42debd39-d410-43a3-8b82-ed8f8e24dbb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:18.784162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42debd39-d410-43a3-8b82-ed8f8e24dbb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:18.787981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42debd39-d410-43a3-8b82-ed8f8e24dbb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:18.788865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42debd39-d410-43a3-8b82-ed8f8e24dbb8 admin admin] PATH_INFO: `/v3/projects/a4ec12eace284a28a8cf620f04718707/users/ff49582d0f2747eab23ec79b63640e4b/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:18.794147 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42debd39-d410-43a3-8b82-ed8f8e24dbb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4ec12eace284a28a8cf620f04718707, user_id=ff49582d0f2747eab23ec79b63640e4b, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:18.802680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42debd39-d410-43a3-8b82-ed8f8e24dbb8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:18.826581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 853/1714] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:18 2022] PUT /identity/v3/projects/a4ec12eace284a28a8cf620f04718707/users/ff49582d0f2747eab23ec79b63640e4b/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:18.832813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757c7be0-b56f-4fd6-aa65-ef76b5fd6b12 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:18.833558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757c7be0-b56f-4fd6-aa65-ef76b5fd6b12 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:18.834425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757c7be0-b56f-4fd6-aa65-ef76b5fd6b12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:18.901257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-757c7be0-b56f-4fd6-aa65-ef76b5fd6b12 None None] MFA Rules not processed for user `5184f8f717d447cfbb7bab80d4cf000b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:18.902501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-757c7be0-b56f-4fd6-aa65-ef76b5fd6b12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:18.912290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57eea38c-936c-43e3-8720-97a3704d6c5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:18.912948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eea38c-936c-43e3-8720-97a3704d6c5b service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:18.913215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eea38c-936c-43e3-8720-97a3704d6c5b service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:18.913486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57eea38c-936c-43e3-8720-97a3704d6c5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:18.927085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:18.937189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57eea38c-936c-43e3-8720-97a3704d6c5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:18.939408 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57eea38c-936c-43e3-8720-97a3704d6c5b service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:18.940958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:18.967921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 862/1715] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:18.969144 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb0e508-2698-4c5a-ad05-c6877505325e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:18.969429 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb0e508-2698-4c5a-ad05-c6877505325e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:18.969698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb0e508-2698-4c5a-ad05-c6877505325e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:19.066460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7fb0e508-2698-4c5a-ad05-c6877505325e None None] MFA Rules not processed for user `ff49582d0f2747eab23ec79b63640e4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:19.076812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7fb0e508-2698-4c5a-ad05-c6877505325e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:19.095085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 854/1716] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:18 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:19.096055 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757c7be0-b56f-4fd6-aa65-ef76b5fd6b12 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:19.099981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-757c7be0-b56f-4fd6-aa65-ef76b5fd6b12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:19.256162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 863/1717] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:18 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:19.257186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb0e508-2698-4c5a-ad05-c6877505325e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:19.260258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7fb0e508-2698-4c5a-ad05-c6877505325e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:19.266901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757c7be0-b56f-4fd6-aa65-ef76b5fd6b12 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:19.282019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ba98ea-5ef8-473e-9f1b-77b52f5893c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:19.282921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ba98ea-5ef8-473e-9f1b-77b52f5893c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:19.283336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ba98ea-5ef8-473e-9f1b-77b52f5893c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:19.283718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ba98ea-5ef8-473e-9f1b-77b52f5893c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:19.284969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ba98ea-5ef8-473e-9f1b-77b52f5893c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:19.287181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ba98ea-5ef8-473e-9f1b-77b52f5893c3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:19.352198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 855/1718] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:19 2022] POST /identity/v3/projects => generated 371 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:19.353968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ba98ea-5ef8-473e-9f1b-77b52f5893c3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:19.365047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb0e508-2698-4c5a-ad05-c6877505325e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:19.497993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ba98ea-5ef8-473e-9f1b-77b52f5893c3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:19.507072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bc2ea4-69df-43d5-b7f9-61189f0ddbfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:19.508429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc2ea4-69df-43d5-b7f9-61189f0ddbfa admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:19.509950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc2ea4-69df-43d5-b7f9-61189f0ddbfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:19.510366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc2ea4-69df-43d5-b7f9-61189f0ddbfa admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:19.511225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc2ea4-69df-43d5-b7f9-61189f0ddbfa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:19.513426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc2ea4-69df-43d5-b7f9-61189f0ddbfa admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:19.521329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109610d9-bb3a-4f3c-9f53-f52667c0e26f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:19.522055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109610d9-bb3a-4f3c-9f53-f52667c0e26f service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:19.523596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109610d9-bb3a-4f3c-9f53-f52667c0e26f service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:19.524007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109610d9-bb3a-4f3c-9f53-f52667c0e26f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:19.551620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 856/1719] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:19 2022] POST /identity/v3/users => generated 292 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:19.553279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bc2ea4-69df-43d5-b7f9-61189f0ddbfa admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:19.555637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-78bc2ea4-69df-43d5-b7f9-61189f0ddbfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:19.574212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bc2ea4-69df-43d5-b7f9-61189f0ddbfa admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:19.580789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:19.582025 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:19.582448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:19.582863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] PATH_INFO: `/v3/users/8b7f2ed50e784c34bd0e8db5245facac` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:19.584286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b7f2ed50e784c34bd0e8db5245facac)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:19.586494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:19.640159 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:19.683315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b7f2ed50e784c34bd0e8db5245facac'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:19.693506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:19.699596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b7f2ed50e784c34bd0e8db5245facac'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:19.700702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b7f2ed50e784c34bd0e8db5245facac'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:19.706724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b7f2ed50e784c34bd0e8db5245facac'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:19.716651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109610d9-bb3a-4f3c-9f53-f52667c0e26f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:19.718502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109610d9-bb3a-4f3c-9f53-f52667c0e26f service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:19.727436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b7f2ed50e784c34bd0e8db5245facac'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:19.742291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 864/1720] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:19 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 485 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:19.745675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109610d9-bb3a-4f3c-9f53-f52667c0e26f service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:19.767737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109610d9-bb3a-4f3c-9f53-f52667c0e26f service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:19.776563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 857/1721] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:19 2022] DELETE /identity/v3/users/8b7f2ed50e784c34bd0e8db5245facac => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:19.902992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8928f837-410e-4fd8-8501-3b16385c484c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:19.904811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8928f837-410e-4fd8-8501-3b16385c484c admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:19.905988 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8928f837-410e-4fd8-8501-3b16385c484c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:19.907638 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8928f837-410e-4fd8-8501-3b16385c484c admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:19.908509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8928f837-410e-4fd8-8501-3b16385c484c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:19.910533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8928f837-410e-4fd8-8501-3b16385c484c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:19.929766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 865/1722] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:19 2022] GET /identity/v3/roles => generated 1420 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:19.930379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:19.932628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:19.952650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8928f837-410e-4fd8-8501-3b16385c484c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:19.954826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8928f837-410e-4fd8-8501-3b16385c484c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:19.955349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32272c67-97ed-4c8f-93d1-6bfe4341fa63 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:19.969215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7d6c05-32cd-4fa5-bf5a-cd08f39b971d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:19.970012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7d6c05-32cd-4fa5-bf5a-cd08f39b971d admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:19.970298 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7d6c05-32cd-4fa5-bf5a-cd08f39b971d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:19.970574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7d6c05-32cd-4fa5-bf5a-cd08f39b971d admin admin] PATH_INFO: `/v3/projects/b55aac0e665b498ab290c912d8bb6526/users/3c9ec468ab4c477dab5a08cd8c6d80ac/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:20.011077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7d6c05-32cd-4fa5-bf5a-cd08f39b971d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b55aac0e665b498ab290c912d8bb6526, user_id=3c9ec468ab4c477dab5a08cd8c6d80ac, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:20.017859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7d6c05-32cd-4fa5-bf5a-cd08f39b971d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:20.033774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8928f837-410e-4fd8-8501-3b16385c484c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:20.043414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dfc633-0dd9-40e9-877a-474cd3f64616 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:20.044321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dfc633-0dd9-40e9-877a-474cd3f64616 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:20.044697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dfc633-0dd9-40e9-877a-474cd3f64616 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:20.045081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dfc633-0dd9-40e9-877a-474cd3f64616 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:20.047336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-63dfc633-0dd9-40e9-877a-474cd3f64616 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:20.048914 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 858/1723] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:19 2022] PUT /identity/v3/projects/b55aac0e665b498ab290c912d8bb6526/users/3c9ec468ab4c477dab5a08cd8c6d80ac/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:20.054960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7d6c05-32cd-4fa5-bf5a-cd08f39b971d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:20.144943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dfc633-0dd9-40e9-877a-474cd3f64616 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:20.147199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dfc633-0dd9-40e9-877a-474cd3f64616 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:20.173098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 866/1724] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:19 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:20.205652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7d6c05-32cd-4fa5-bf5a-cd08f39b971d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:20.220992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10991b45-d1cb-4917-b0ae-38cc74caa48c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:20.222946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10991b45-d1cb-4917-b0ae-38cc74caa48c admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:20.223302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10991b45-d1cb-4917-b0ae-38cc74caa48c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:20.223652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10991b45-d1cb-4917-b0ae-38cc74caa48c admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:20.228446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10991b45-d1cb-4917-b0ae-38cc74caa48c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:20.231216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10991b45-d1cb-4917-b0ae-38cc74caa48c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:20.258768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dfc633-0dd9-40e9-877a-474cd3f64616 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:20.272181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 859/1725] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:20 2022] GET /identity/v3/roles => generated 1420 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:20.275976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dfc633-0dd9-40e9-877a-474cd3f64616 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:20.289920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ed146f-4eee-47db-ab56-48c7ad1bb9a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:20.290566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ed146f-4eee-47db-ab56-48c7ad1bb9a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:20.290856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ed146f-4eee-47db-ab56-48c7ad1bb9a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:20.291137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ed146f-4eee-47db-ab56-48c7ad1bb9a4 admin admin] PATH_INFO: `/v3/projects/b55aac0e665b498ab290c912d8bb6526/users/3c9ec468ab4c477dab5a08cd8c6d80ac/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:20.301962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ed146f-4eee-47db-ab56-48c7ad1bb9a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b55aac0e665b498ab290c912d8bb6526, user_id=3c9ec468ab4c477dab5a08cd8c6d80ac, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:20.313832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ed146f-4eee-47db-ab56-48c7ad1bb9a4 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:20.351642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 867/1726] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:20 2022] PUT /identity/v3/projects/b55aac0e665b498ab290c912d8bb6526/users/3c9ec468ab4c477dab5a08cd8c6d80ac/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:20.640384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10991b45-d1cb-4917-b0ae-38cc74caa48c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:20.644134 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-10991b45-d1cb-4917-b0ae-38cc74caa48c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:20.671674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10991b45-d1cb-4917-b0ae-38cc74caa48c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:20.827445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:20.828273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:20.829448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:20.829872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] PATH_INFO: `/v3/projects/e95f9dadfbb744668231b3ada7960483` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:20.834841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e95f9dadfbb744668231b3ada7960483)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:20.840093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:20.873335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194da96b-608f-4a46-930f-077b3d5cc818 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:20.873821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194da96b-608f-4a46-930f-077b3d5cc818 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:20.874220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194da96b-608f-4a46-930f-077b3d5cc818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:20.918870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:20.952614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-194da96b-608f-4a46-930f-077b3d5cc818 None None] MFA Rules not processed for user `73c365df91254de6a9217dab0280eff3`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:20.954042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-194da96b-608f-4a46-930f-077b3d5cc818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:20.970846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e95f9dadfbb744668231b3ada7960483', 'request_id': 'req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:20.987005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e95f9dadfbb744668231b3ada7960483', 'request_id': 'req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:20.988914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:21.008472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e95f9dadfbb744668231b3ada7960483', 'request_id': 'req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:21.054535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] The token cache is being invalidate because project e95f9dadfbb744668231b3ada7960483 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:21.054858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:21.058540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:21.068421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 860/1727] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:20 2022] DELETE /identity/v3/projects/e95f9dadfbb744668231b3ada7960483 => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:21.070570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:21.072574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:21.123855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 868/1728] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:20 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:21.126059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90a2324-de35-4ae8-9bae-6445b61b681a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:21.126432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90a2324-de35-4ae8-9bae-6445b61b681a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:21.126778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90a2324-de35-4ae8-9bae-6445b61b681a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:21.167987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-f90a2324-de35-4ae8-9bae-6445b61b681a None None] MFA Rules not processed for user `73c365df91254de6a9217dab0280eff3`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:21.169537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f90a2324-de35-4ae8-9bae-6445b61b681a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:21.195244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 869/1729] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:21 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:21.198296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90a2324-de35-4ae8-9bae-6445b61b681a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:21.207887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f90a2324-de35-4ae8-9bae-6445b61b681a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:21.236426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2305a73-16f8-4d72-9fed-fdd8a3bc3cc6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:21.254075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:21.254888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:21.255175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:21.255551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] PATH_INFO: `/v3/users/d149dbf5ef87428ba6009e1d16a0e6ae` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:21.257534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d149dbf5ef87428ba6009e1d16a0e6ae)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:21.260851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:21.345429 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90a2324-de35-4ae8-9bae-6445b61b681a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:21.353184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d27044-4635-4da5-b8da-1feeef4d1ee4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:21.354368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d27044-4635-4da5-b8da-1feeef4d1ee4 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:21.354850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d27044-4635-4da5-b8da-1feeef4d1ee4 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:21.356502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d27044-4635-4da5-b8da-1feeef4d1ee4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:21.363212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd149dbf5ef87428ba6009e1d16a0e6ae'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:21.370881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d27044-4635-4da5-b8da-1feeef4d1ee4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:21.372626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d27044-4635-4da5-b8da-1feeef4d1ee4 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:21.385252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd149dbf5ef87428ba6009e1d16a0e6ae'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:21.386134 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd149dbf5ef87428ba6009e1d16a0e6ae'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:21.390368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd149dbf5ef87428ba6009e1d16a0e6ae'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:21.399250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 870/1730] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:21 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:21.433197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e557c197-6f38-4007-909d-8e1dd7d41e45 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd149dbf5ef87428ba6009e1d16a0e6ae'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:21.465984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 861/1731] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:21 2022] DELETE /identity/v3/users/d149dbf5ef87428ba6009e1d16a0e6ae => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:21.523382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d27044-4635-4da5-b8da-1feeef4d1ee4 service nova] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:21.525064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-58d27044-4635-4da5-b8da-1feeef4d1ee4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:21.576397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d27044-4635-4da5-b8da-1feeef4d1ee4 service nova] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:21.754177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931029b-6f07-44a1-9a9a-fb4c8c1e3ab8 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:21.754177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931029b-6f07-44a1-9a9a-fb4c8c1e3ab8 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:21.755334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931029b-6f07-44a1-9a9a-fb4c8c1e3ab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:21.767929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:21.770223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:21.770519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:21.770793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] PATH_INFO: `/v3/projects/f6ae179ea71e41a79a5858d177458a1e` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:21.778224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6ae179ea71e41a79a5858d177458a1e)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:21.783553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:21.858053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1931029b-6f07-44a1-9a9a-fb4c8c1e3ab8 None None] MFA Rules not processed for user `fb54ba1355004e14b8dc43cf3c028e67`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:21.859966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1931029b-6f07-44a1-9a9a-fb4c8c1e3ab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:21.893860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:21.911846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:21.950308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6ae179ea71e41a79a5858d177458a1e', 'request_id': 'req-358097e8-29cd-4a82-b923-7768e30e3fd2'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:21.976468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6ae179ea71e41a79a5858d177458a1e', 'request_id': 'req-358097e8-29cd-4a82-b923-7768e30e3fd2'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:21.992069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6ae179ea71e41a79a5858d177458a1e', 'request_id': 'req-358097e8-29cd-4a82-b923-7768e30e3fd2'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:22.040601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] The token cache is being invalidate because project f6ae179ea71e41a79a5858d177458a1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:22.045700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:22.048800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-358097e8-29cd-4a82-b923-7768e30e3fd2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:22.055230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 871/1732] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:21 2022] DELETE /identity/v3/projects/f6ae179ea71e41a79a5858d177458a1e => generated 0 bytes in 530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:22.057575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 862/1733] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:21 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:22.058492 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931029b-6f07-44a1-9a9a-fb4c8c1e3ab8 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:22.067583 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890146df-6be7-473d-a91c-5fbc178415b6 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:22.067947 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890146df-6be7-473d-a91c-5fbc178415b6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:22.068279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890146df-6be7-473d-a91c-5fbc178415b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:22.071718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1931029b-6f07-44a1-9a9a-fb4c8c1e3ab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:22.114558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931029b-6f07-44a1-9a9a-fb4c8c1e3ab8 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:22.146031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-890146df-6be7-473d-a91c-5fbc178415b6 None None] MFA Rules not processed for user `bc62747b8aab40d9bff9d853b9b73199`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:22.157061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-890146df-6be7-473d-a91c-5fbc178415b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:22.300262 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:22.302065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:22.302667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:22.303314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] PATH_INFO: `/v3/users/e99e0aa6af854e8aaa7c263c42814e77` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:22.304743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] RBAC: Authorizing `identity:delete_user(user_id=e99e0aa6af854e8aaa7c263c42814e77)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:22.313973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:22.340432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 872/1734] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:22 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:22.350116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633bc7fd-3af1-45a8-9b27-4935bb25ab3b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:22.354077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633bc7fd-3af1-45a8-9b27-4935bb25ab3b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:22.355215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633bc7fd-3af1-45a8-9b27-4935bb25ab3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:22.405081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-633bc7fd-3af1-45a8-9b27-4935bb25ab3b None None] MFA Rules not processed for user `fb54ba1355004e14b8dc43cf3c028e67`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:22.408336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-633bc7fd-3af1-45a8-9b27-4935bb25ab3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:22.471992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e99e0aa6af854e8aaa7c263c42814e77'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:22.484800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e99e0aa6af854e8aaa7c263c42814e77'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:22.486428 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e99e0aa6af854e8aaa7c263c42814e77'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:22.487789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e99e0aa6af854e8aaa7c263c42814e77'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:22.524332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e99e0aa6af854e8aaa7c263c42814e77'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:22.555154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 863/1735] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:22 2022] DELETE /identity/v3/users/e99e0aa6af854e8aaa7c263c42814e77 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:22.557487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:22.559121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:22.628728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 873/1736] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:22 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:22.629721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633bc7fd-3af1-45a8-9b27-4935bb25ab3b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:22.633438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-633bc7fd-3af1-45a8-9b27-4935bb25ab3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:22.640808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:22.719794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633bc7fd-3af1-45a8-9b27-4935bb25ab3b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:22.756321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a3b173-7ad7-4e06-ad90-97ea005d4eab admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:22.778610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a893b25-8528-4aeb-a409-347992708243 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:22.779671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a893b25-8528-4aeb-a409-347992708243 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:22.779671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a893b25-8528-4aeb-a409-347992708243 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:22.779927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a893b25-8528-4aeb-a409-347992708243 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:22.783952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3a893b25-8528-4aeb-a409-347992708243 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:22.924772 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:22.951468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:22.952132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:22.952437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:22.952824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a893b25-8528-4aeb-a409-347992708243 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:22.955094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a893b25-8528-4aeb-a409-347992708243 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:22.955549 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] PATH_INFO: `/v3/projects/2135e08f67b54af6ba37caf73a0d2629` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:22.960609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2135e08f67b54af6ba37caf73a0d2629)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:22.966678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:22.969756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 864/1737] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:22 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:23.003524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a893b25-8528-4aeb-a409-347992708243 service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:23.005657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3a893b25-8528-4aeb-a409-347992708243 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:23.047783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a893b25-8528-4aeb-a409-347992708243 service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:23.080300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2135e08f67b54af6ba37caf73a0d2629', 'request_id': 'req-7c9ffa33-bb84-496a-8c86-c9f444271905'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:23.083796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:23.088054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.088399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.088869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] PATH_INFO: `/v3/projects/977a345fcfff467f85889ab3d4a63aef` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.091600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] RBAC: Authorizing `identity:delete_project(project_id=977a345fcfff467f85889ab3d4a63aef)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:23.096751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:23.108385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2135e08f67b54af6ba37caf73a0d2629', 'request_id': 'req-7c9ffa33-bb84-496a-8c86-c9f444271905'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:23.119771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2135e08f67b54af6ba37caf73a0d2629', 'request_id': 'req-7c9ffa33-bb84-496a-8c86-c9f444271905'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:23.142278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] The token cache is being invalidate because project 2135e08f67b54af6ba37caf73a0d2629 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:23.142607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:23.147644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7c9ffa33-bb84-496a-8c86-c9f444271905 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:23.155094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 874/1738] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:22 2022] DELETE /identity/v3/projects/2135e08f67b54af6ba37caf73a0d2629 => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:23.157596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da883e-5623-4a0b-8894-e590da101ca4 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.157976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da883e-5623-4a0b-8894-e590da101ca4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.158906 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da883e-5623-4a0b-8894-e590da101ca4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.215531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-a2da883e-5623-4a0b-8894-e590da101ca4 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:23.232323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a2da883e-5623-4a0b-8894-e590da101ca4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:23.242899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '977a345fcfff467f85889ab3d4a63aef', 'request_id': 'req-688af1fe-773b-4c35-ab75-8dc901377131'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:23.268714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '977a345fcfff467f85889ab3d4a63aef', 'request_id': 'req-688af1fe-773b-4c35-ab75-8dc901377131'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:23.294137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '977a345fcfff467f85889ab3d4a63aef', 'request_id': 'req-688af1fe-773b-4c35-ab75-8dc901377131'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:23.342578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] The token cache is being invalidate because project 977a345fcfff467f85889ab3d4a63aef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:23.342926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:23.347252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-688af1fe-773b-4c35-ab75-8dc901377131 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:23.347853 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 875/1739] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:23.348959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2da883e-5623-4a0b-8894-e590da101ca4 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:23.350510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a2da883e-5623-4a0b-8894-e590da101ca4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:23.351526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 865/1740] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:23 2022] DELETE /identity/v3/projects/977a345fcfff467f85889ab3d4a63aef => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:23.353587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae402607-ae9f-450c-90f1-c4749712006d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.354007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae402607-ae9f-450c-90f1-c4749712006d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.354286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae402607-ae9f-450c-90f1-c4749712006d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.434483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ae402607-ae9f-450c-90f1-c4749712006d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:23.436278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ae402607-ae9f-450c-90f1-c4749712006d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:23.457396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 866/1741] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:23.458828 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff5885-cf83-4ee2-ac05-fee20d7ca058 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.459239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff5885-cf83-4ee2-ac05-fee20d7ca058 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.459530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff5885-cf83-4ee2-ac05-fee20d7ca058 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.486241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-43ff5885-cf83-4ee2-ac05-fee20d7ca058 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:23.487611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-43ff5885-cf83-4ee2-ac05-fee20d7ca058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:23.499505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 867/1742] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:23.500386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff5885-cf83-4ee2-ac05-fee20d7ca058 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:23.521752 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2da883e-5623-4a0b-8894-e590da101ca4 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:23.532129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f05626-51e3-45dc-bec7-eca06436d7bd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:23.532808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f05626-51e3-45dc-bec7-eca06436d7bd service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.533093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f05626-51e3-45dc-bec7-eca06436d7bd service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.533369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f05626-51e3-45dc-bec7-eca06436d7bd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.543504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff5885-cf83-4ee2-ac05-fee20d7ca058 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:23.547512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b7f05626-51e3-45dc-bec7-eca06436d7bd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:23.556927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46857243-a4e4-4772-89a9-645e6203744a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:23.558115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46857243-a4e4-4772-89a9-645e6203744a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.559728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46857243-a4e4-4772-89a9-645e6203744a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.561137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46857243-a4e4-4772-89a9-645e6203744a admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.564011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46857243-a4e4-4772-89a9-645e6203744a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:23.564552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46857243-a4e4-4772-89a9-645e6203744a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:23.567572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46857243-a4e4-4772-89a9-645e6203744a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:23.606134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:23.634977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 868/1743] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:23.637420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca081555-6872-40bd-852e-11b907b95c73 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.640603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca081555-6872-40bd-852e-11b907b95c73 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.642529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca081555-6872-40bd-852e-11b907b95c73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.715811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ca081555-6872-40bd-852e-11b907b95c73 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:23.717310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ca081555-6872-40bd-852e-11b907b95c73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:23.727364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f05626-51e3-45dc-bec7-eca06436d7bd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:23.732124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 869/1744] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:23.733744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaed8a4-c2cd-499c-880f-36aad36e7ad1 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.734620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaed8a4-c2cd-499c-880f-36aad36e7ad1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.734620 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaed8a4-c2cd-499c-880f-36aad36e7ad1 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.735986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f05626-51e3-45dc-bec7-eca06436d7bd service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:23.737045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 870/1745] 10.222.0.25 () {54 vars in 912 bytes} [Mon Sep 26 01:36:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:23.737851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaed8a4-c2cd-499c-880f-36aad36e7ad1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:23.782868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaed8a4-c2cd-499c-880f-36aad36e7ad1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:23.799145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 876/1746] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:23 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:23.810073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30348456-70b4-4084-9d02-b68e11328d13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:23.810760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30348456-70b4-4084-9d02-b68e11328d13 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.811038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30348456-70b4-4084-9d02-b68e11328d13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.811360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30348456-70b4-4084-9d02-b68e11328d13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.812931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30348456-70b4-4084-9d02-b68e11328d13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:23.819767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30348456-70b4-4084-9d02-b68e11328d13 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:23.822786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f05626-51e3-45dc-bec7-eca06436d7bd service nova] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:23.872238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f05626-51e3-45dc-bec7-eca06436d7bd service nova] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:23.892207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9e28cb-ae37-4bcd-a0e2-496ae44ee8dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:23.892882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9e28cb-ae37-4bcd-a0e2-496ae44ee8dc admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.893167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9e28cb-ae37-4bcd-a0e2-496ae44ee8dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.893443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9e28cb-ae37-4bcd-a0e2-496ae44ee8dc admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.894104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9e28cb-ae37-4bcd-a0e2-496ae44ee8dc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:23.894422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9e28cb-ae37-4bcd-a0e2-496ae44ee8dc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:23.896049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9e28cb-ae37-4bcd-a0e2-496ae44ee8dc admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:23.907097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 871/1747] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:23 2022] POST /identity/v3/projects => generated 385 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:23.907097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ff11c-9b38-495b-b3bc-3f241c736dcd None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:23.907097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ff11c-9b38-495b-b3bc-3f241c736dcd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:23.907097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ff11c-9b38-495b-b3bc-3f241c736dcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:23.945313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:23.958818 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 877/1748] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:23.969245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9e28cb-ae37-4bcd-a0e2-496ae44ee8dc admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:24.009687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9e28cb-ae37-4bcd-a0e2-496ae44ee8dc admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:24.085712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-121ff11c-9b38-495b-b3bc-3f241c736dcd None None] MFA Rules not processed for user `0478ebab37fb4f488eef84f7a241e206`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:24.087097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-121ff11c-9b38-495b-b3bc-3f241c736dcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:24.237554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b2a642-a84f-4df9-8fd6-e5e74a084a73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:24.238404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b2a642-a84f-4df9-8fd6-e5e74a084a73 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:24.238766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b2a642-a84f-4df9-8fd6-e5e74a084a73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:24.239044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b2a642-a84f-4df9-8fd6-e5e74a084a73 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:24.240943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b2a642-a84f-4df9-8fd6-e5e74a084a73 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:24.247773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b2a642-a84f-4df9-8fd6-e5e74a084a73 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:24.295185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 878/1749] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:23 2022] POST /identity/v3/users => generated 293 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:24.297458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b2a642-a84f-4df9-8fd6-e5e74a084a73 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:24.306842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 872/1750] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:36:23 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 399 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:24.307889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121ff11c-9b38-495b-b3bc-3f241c736dcd None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:24.320951 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b2a642-a84f-4df9-8fd6-e5e74a084a73 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:24.333630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d2171b-408c-4bd1-ac94-aeb5576f360c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:24.334525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2171b-408c-4bd1-ac94-aeb5576f360c admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:24.335127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2171b-408c-4bd1-ac94-aeb5576f360c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:24.335880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2171b-408c-4bd1-ac94-aeb5576f360c admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:24.338221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d2171b-408c-4bd1-ac94-aeb5576f360c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:24.340785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d2171b-408c-4bd1-ac94-aeb5576f360c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:24.342375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121ff11c-9b38-495b-b3bc-3f241c736dcd None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:24.365723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b3f27d-cf02-4f6b-a57d-3fad1b802a1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:24.373166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b3f27d-cf02-4f6b-a57d-3fad1b802a1e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:24.373608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b3f27d-cf02-4f6b-a57d-3fad1b802a1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:24.373983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b3f27d-cf02-4f6b-a57d-3fad1b802a1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:24.374918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b3f27d-cf02-4f6b-a57d-3fad1b802a1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:24.377013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b3f27d-cf02-4f6b-a57d-3fad1b802a1e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:24.403618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 873/1751] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:24 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:24.404948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b3f27d-cf02-4f6b-a57d-3fad1b802a1e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:24.406302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 879/1752] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:24 2022] POST /identity/v3/projects => generated 381 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:24.407095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d2171b-408c-4bd1-ac94-aeb5576f360c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:24.411250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e0b3f27d-cf02-4f6b-a57d-3fad1b802a1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:24.443701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b3f27d-cf02-4f6b-a57d-3fad1b802a1e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:24.551186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef866656-889c-4ccd-9630-cd4441799f25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:24.570810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef866656-889c-4ccd-9630-cd4441799f25 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:24.577002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef866656-889c-4ccd-9630-cd4441799f25 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:24.577988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef866656-889c-4ccd-9630-cd4441799f25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:24.612541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef866656-889c-4ccd-9630-cd4441799f25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:24.614064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef866656-889c-4ccd-9630-cd4441799f25 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:24.624217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d2171b-408c-4bd1-ac94-aeb5576f360c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:24.643736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 874/1753] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:24 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:24.644617 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef866656-889c-4ccd-9630-cd4441799f25 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:24.647684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e040a7a-aefa-40b8-b48a-1641366782f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:24.648325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e040a7a-aefa-40b8-b48a-1641366782f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:24.648629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e040a7a-aefa-40b8-b48a-1641366782f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:24.648900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e040a7a-aefa-40b8-b48a-1641366782f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:24.649662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e040a7a-aefa-40b8-b48a-1641366782f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:24.651334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e040a7a-aefa-40b8-b48a-1641366782f3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:24.685546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef866656-889c-4ccd-9630-cd4441799f25 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:24.687672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 880/1754] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:24 2022] GET /identity/v3/roles => generated 1420 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:24.689117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f11a592-9919-423b-98ad-22a2d4e0526f None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:24.689402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f11a592-9919-423b-98ad-22a2d4e0526f None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:24.689674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f11a592-9919-423b-98ad-22a2d4e0526f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:24.711317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c737fe-42d7-4346-a423-df558cb2e831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:24.711991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c737fe-42d7-4346-a423-df558cb2e831 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:24.712284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c737fe-42d7-4346-a423-df558cb2e831 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:24.712825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c737fe-42d7-4346-a423-df558cb2e831 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:24.713664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c737fe-42d7-4346-a423-df558cb2e831 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:24.715603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c737fe-42d7-4346-a423-df558cb2e831 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:24.742531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-1f11a592-9919-423b-98ad-22a2d4e0526f None None] MFA Rules not processed for user `06043e862a394764a58003f205cbb484`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:24.745276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1f11a592-9919-423b-98ad-22a2d4e0526f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:24.753014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:24.768972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 875/1755] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:24 2022] POST /identity/v3/users => generated 291 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:24.770197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c737fe-42d7-4346-a423-df558cb2e831 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:24.781758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c737fe-42d7-4346-a423-df558cb2e831 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:24.790673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfe0979-70c8-4ef6-bb31-52f3e36fc58c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:24.791390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfe0979-70c8-4ef6-bb31-52f3e36fc58c admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:24.792233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfe0979-70c8-4ef6-bb31-52f3e36fc58c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:24.794131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfe0979-70c8-4ef6-bb31-52f3e36fc58c admin admin] PATH_INFO: `/v3/projects/3fe4653e43a14bb7b7bbc2f6cbd008cb/users/59aaa75b0a324985aca9cb823ce1188c/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:24.830269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfe0979-70c8-4ef6-bb31-52f3e36fc58c admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fe4653e43a14bb7b7bbc2f6cbd008cb, user_id=59aaa75b0a324985aca9cb823ce1188c, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:24.835934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dfe0979-70c8-4ef6-bb31-52f3e36fc58c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:24.864170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 881/1756] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:24 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:24.865672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 876/1757] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:24 2022] PUT /identity/v3/projects/3fe4653e43a14bb7b7bbc2f6cbd008cb/users/59aaa75b0a324985aca9cb823ce1188c/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:24.866289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f11a592-9919-423b-98ad-22a2d4e0526f None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:24.875636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aba373b-4740-49e1-9eb0-ad2285dbd166 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:24.875922 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aba373b-4740-49e1-9eb0-ad2285dbd166 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:24.876190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aba373b-4740-49e1-9eb0-ad2285dbd166 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:24.905665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f11a592-9919-423b-98ad-22a2d4e0526f None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:24.950618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4aba373b-4740-49e1-9eb0-ad2285dbd166 None None] MFA Rules not processed for user `59aaa75b0a324985aca9cb823ce1188c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:24.951868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4aba373b-4740-49e1-9eb0-ad2285dbd166 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:25.054130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:25.055505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d25a6e8-9a89-40f3-84dc-2c92712a28be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:25.056742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d25a6e8-9a89-40f3-84dc-2c92712a28be admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:25.057155 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d25a6e8-9a89-40f3-84dc-2c92712a28be admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:25.057488 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d25a6e8-9a89-40f3-84dc-2c92712a28be admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:25.058157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d25a6e8-9a89-40f3-84dc-2c92712a28be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:25.061312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d25a6e8-9a89-40f3-84dc-2c92712a28be admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:25.100952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 882/1758] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:24 2022] GET /identity/v3/roles => generated 1420 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:25.102381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b61b5-150b-4ad3-b536-8f4ff5b4aea4 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:25.102898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b61b5-150b-4ad3-b536-8f4ff5b4aea4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:25.103313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b61b5-150b-4ad3-b536-8f4ff5b4aea4 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:25.110637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 883/1759] 10.222.0.25 () {54 vars in 912 bytes} [Mon Sep 26 01:36:25 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:25.111971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac8a724-1fee-45bb-9dea-5dcf6355335b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:25.112270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac8a724-1fee-45bb-9dea-5dcf6355335b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:25.112563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac8a724-1fee-45bb-9dea-5dcf6355335b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:25.126690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 877/1760] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:24 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:25.131421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aba373b-4740-49e1-9eb0-ad2285dbd166 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:25.151959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-cac8a724-1fee-45bb-9dea-5dcf6355335b None None] MFA Rules not processed for user `3c9ec468ab4c477dab5a08cd8c6d80ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:25.153236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-cac8a724-1fee-45bb-9dea-5dcf6355335b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:25.176284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aba373b-4740-49e1-9eb0-ad2285dbd166 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:25.185956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ade554-d5a5-4f19-930d-d240a53326be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:25.186904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ade554-d5a5-4f19-930d-d240a53326be admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:25.186904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ade554-d5a5-4f19-930d-d240a53326be admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:25.187085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ade554-d5a5-4f19-930d-d240a53326be admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:25.187738 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ade554-d5a5-4f19-930d-d240a53326be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:25.189372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ade554-d5a5-4f19-930d-d240a53326be admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:25.217000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 878/1761] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:25 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:25.220061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47503cdb-b883-4f0d-83c8-221e7357e63d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:25.225399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47503cdb-b883-4f0d-83c8-221e7357e63d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:25.225621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47503cdb-b883-4f0d-83c8-221e7357e63d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:25.274895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-47503cdb-b883-4f0d-83c8-221e7357e63d None None] MFA Rules not processed for user `0478ebab37fb4f488eef84f7a241e206`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:25.276150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-47503cdb-b883-4f0d-83c8-221e7357e63d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:25.351871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 884/1762] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:25 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:25.357303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac8a724-1fee-45bb-9dea-5dcf6355335b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:25.385530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac8a724-1fee-45bb-9dea-5dcf6355335b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:25.392656 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4a34c4-5d1a-4350-9523-a7ed1323ddc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:25.393391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4a34c4-5d1a-4350-9523-a7ed1323ddc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:25.393672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4a34c4-5d1a-4350-9523-a7ed1323ddc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:25.393940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4a34c4-5d1a-4350-9523-a7ed1323ddc4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:25.394900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4a34c4-5d1a-4350-9523-a7ed1323ddc4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:25.402328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4a34c4-5d1a-4350-9523-a7ed1323ddc4 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:25.405506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 879/1763] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:36:25 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:25.406669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47503cdb-b883-4f0d-83c8-221e7357e63d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:25.420841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47503cdb-b883-4f0d-83c8-221e7357e63d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:25.425809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56472b11-6ba6-4d65-868e-97d4e84e7a48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:25.426749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56472b11-6ba6-4d65-868e-97d4e84e7a48 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:25.427201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56472b11-6ba6-4d65-868e-97d4e84e7a48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:25.427850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56472b11-6ba6-4d65-868e-97d4e84e7a48 admin admin] PATH_INFO: `/v3/projects/db5fce1fe9214c48b0aff3c0de81a8b5/users/7eea683669f846c08e9f9751270f24fa/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:25.486722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 885/1764] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:25 2022] POST /identity/v3/projects => generated 387 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:25.487579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4a34c4-5d1a-4350-9523-a7ed1323ddc4 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:25.491445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56472b11-6ba6-4d65-868e-97d4e84e7a48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db5fce1fe9214c48b0aff3c0de81a8b5, user_id=7eea683669f846c08e9f9751270f24fa, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:25.496060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56472b11-6ba6-4d65-868e-97d4e84e7a48 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:25.500129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6b4a34c4-5d1a-4350-9523-a7ed1323ddc4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:25.518593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:25.528217 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 880/1765] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:25 2022] PUT /identity/v3/projects/db5fce1fe9214c48b0aff3c0de81a8b5/users/7eea683669f846c08e9f9751270f24fa/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:25.529221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56472b11-6ba6-4d65-868e-97d4e84e7a48 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:25.541773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-56472b11-6ba6-4d65-868e-97d4e84e7a48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:25.659304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4a34c4-5d1a-4350-9523-a7ed1323ddc4 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:25.700246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56472b11-6ba6-4d65-868e-97d4e84e7a48 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:25.709092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2a071d-4e07-43b2-a44a-6338a29beef5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:25.709860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2a071d-4e07-43b2-a44a-6338a29beef5 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:25.710151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2a071d-4e07-43b2-a44a-6338a29beef5 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:25.710432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2a071d-4e07-43b2-a44a-6338a29beef5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:25.825900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:25.829196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ad36f8-cbff-4ed0-9ce7-cd45b4b55df7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:25.830707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ad36f8-cbff-4ed0-9ce7-cd45b4b55df7 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:25.831003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ad36f8-cbff-4ed0-9ce7-cd45b4b55df7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:25.831282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ad36f8-cbff-4ed0-9ce7-cd45b4b55df7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:25.834197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b7ad36f8-cbff-4ed0-9ce7-cd45b4b55df7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:25.852256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2a071d-4e07-43b2-a44a-6338a29beef5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:25.861436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2a071d-4e07-43b2-a44a-6338a29beef5 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:25.883294 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 881/1766] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:25 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:25.887379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2a071d-4e07-43b2-a44a-6338a29beef5 service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:26.091440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2a071d-4e07-43b2-a44a-6338a29beef5 service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:26.118782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a8f3f8-c2d7-475c-9073-6ec07b6194b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:26.119696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a8f3f8-c2d7-475c-9073-6ec07b6194b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:26.119983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a8f3f8-c2d7-475c-9073-6ec07b6194b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:26.120295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a8f3f8-c2d7-475c-9073-6ec07b6194b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:26.151744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ad36f8-cbff-4ed0-9ce7-cd45b4b55df7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:26.152708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a8f3f8-c2d7-475c-9073-6ec07b6194b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:26.157391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a8f3f8-c2d7-475c-9073-6ec07b6194b0 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:26.157720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ad36f8-cbff-4ed0-9ce7-cd45b4b55df7 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:26.180329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 886/1767] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:25 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 691 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:26.180329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ad36f8-cbff-4ed0-9ce7-cd45b4b55df7 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:26.198303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 882/1768] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:26.204801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ba4d58-9921-4872-b885-5e05fa8b3cbd None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:26.205524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ba4d58-9921-4872-b885-5e05fa8b3cbd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:26.205524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ba4d58-9921-4872-b885-5e05fa8b3cbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:26.307743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-04ba4d58-9921-4872-b885-5e05fa8b3cbd None None] MFA Rules not processed for user `7eea683669f846c08e9f9751270f24fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:26.310304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-04ba4d58-9921-4872-b885-5e05fa8b3cbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:26.417537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ad36f8-cbff-4ed0-9ce7-cd45b4b55df7 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:26.440520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d253a-5489-4725-96e6-820c573fada6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:26.443716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d253a-5489-4725-96e6-820c573fada6 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:26.443716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d253a-5489-4725-96e6-820c573fada6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:26.443716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d253a-5489-4725-96e6-820c573fada6 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:26.443716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832d253a-5489-4725-96e6-820c573fada6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:26.454944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832d253a-5489-4725-96e6-820c573fada6 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:26.524710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 887/1769] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:26 2022] POST /identity/v3/users => generated 300 bytes in 344 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:26.526015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d253a-5489-4725-96e6-820c573fada6 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:26.534769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-832d253a-5489-4725-96e6-820c573fada6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:26.547272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:26.592915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 883/1770] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:26 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 394 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:26.600118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ba4d58-9921-4872-b885-5e05fa8b3cbd None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:26.636417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ba4d58-9921-4872-b885-5e05fa8b3cbd None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:26.649075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a553e98-14d1-4424-a591-f89a5e9a8762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:26.652496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a553e98-14d1-4424-a591-f89a5e9a8762 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:26.652952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a553e98-14d1-4424-a591-f89a5e9a8762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:26.653392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a553e98-14d1-4424-a591-f89a5e9a8762 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:26.654210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a553e98-14d1-4424-a591-f89a5e9a8762 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:26.659935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a553e98-14d1-4424-a591-f89a5e9a8762 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:26.688706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 884/1771] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:26 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:26.689875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a553e98-14d1-4424-a591-f89a5e9a8762 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:26.736431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a553e98-14d1-4424-a591-f89a5e9a8762 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:26.753472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c0b7ab-c83c-4514-be38-1116df7eb609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:26.758188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c0b7ab-c83c-4514-be38-1116df7eb609 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:26.758467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c0b7ab-c83c-4514-be38-1116df7eb609 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:26.759708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c0b7ab-c83c-4514-be38-1116df7eb609 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:26.763920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c0b7ab-c83c-4514-be38-1116df7eb609 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:26.769639 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c0b7ab-c83c-4514-be38-1116df7eb609 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:26.800093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d253a-5489-4725-96e6-820c573fada6 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:26.814648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11c8ab9-e290-4d60-862b-7c0316e410b1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:26.815105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11c8ab9-e290-4d60-862b-7c0316e410b1 service placement] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:26.815355 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11c8ab9-e290-4d60-862b-7c0316e410b1 service placement] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:26.815623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11c8ab9-e290-4d60-862b-7c0316e410b1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:26.818175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d11c8ab9-e290-4d60-862b-7c0316e410b1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:26.857268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 885/1772] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:26 2022] POST /identity/v3/projects => generated 381 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:26.859987 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c0b7ab-c83c-4514-be38-1116df7eb609 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:26.874914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11c8ab9-e290-4d60-862b-7c0316e410b1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:26.879732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11c8ab9-e290-4d60-862b-7c0316e410b1 service placement] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:26.927786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 888/1773] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:26.931113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11c8ab9-e290-4d60-862b-7c0316e410b1 service placement] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:26.968060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11c8ab9-e290-4d60-862b-7c0316e410b1 service placement] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:26.985858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:27.055012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c0b7ab-c83c-4514-be38-1116df7eb609 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:27.101234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10fa03e-f93e-42fb-a737-58daf1e4ae49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:27.104938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10fa03e-f93e-42fb-a737-58daf1e4ae49 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:27.105568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10fa03e-f93e-42fb-a737-58daf1e4ae49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:27.106963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10fa03e-f93e-42fb-a737-58daf1e4ae49 admin admin] PATH_INFO: `/v3/projects/06294382b1a84a30aad2382c3a7cfcca/users/7d981ab53c564e4c8ecf3adf3af3acd4/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:27.166778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0024e44c-e405-4b93-98eb-9975ec4c8046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:27.168021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0024e44c-e405-4b93-98eb-9975ec4c8046 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:27.168465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0024e44c-e405-4b93-98eb-9975ec4c8046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:27.169201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0024e44c-e405-4b93-98eb-9975ec4c8046 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:27.170148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0024e44c-e405-4b93-98eb-9975ec4c8046 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:27.184237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10fa03e-f93e-42fb-a737-58daf1e4ae49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=06294382b1a84a30aad2382c3a7cfcca, user_id=7d981ab53c564e4c8ecf3adf3af3acd4, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:27.184237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10fa03e-f93e-42fb-a737-58daf1e4ae49 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:27.188824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0024e44c-e405-4b93-98eb-9975ec4c8046 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:27.218963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 886/1774] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:26 2022] PUT /identity/v3/projects/06294382b1a84a30aad2382c3a7cfcca/users/7d981ab53c564e4c8ecf3adf3af3acd4/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:27.221549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10fa03e-f93e-42fb-a737-58daf1e4ae49 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:27.242310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 889/1775] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:26 2022] POST /identity/v3/users => generated 297 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:27.243892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0024e44c-e405-4b93-98eb-9975ec4c8046 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:27.269286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0024e44c-e405-4b93-98eb-9975ec4c8046 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:27.387671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10fa03e-f93e-42fb-a737-58daf1e4ae49 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:27.398467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba64d265-9677-4682-9bfa-13bc31f434dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:27.399094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba64d265-9677-4682-9bfa-13bc31f434dd admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:27.399407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba64d265-9677-4682-9bfa-13bc31f434dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:27.399677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba64d265-9677-4682-9bfa-13bc31f434dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:27.400322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba64d265-9677-4682-9bfa-13bc31f434dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:27.406200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba64d265-9677-4682-9bfa-13bc31f434dd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:27.417169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f1aaa9-d5ab-4a45-a78f-802cd8f1d397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:27.418025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f1aaa9-d5ab-4a45-a78f-802cd8f1d397 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:27.418313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f1aaa9-d5ab-4a45-a78f-802cd8f1d397 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:27.418582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f1aaa9-d5ab-4a45-a78f-802cd8f1d397 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:27.419245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f1aaa9-d5ab-4a45-a78f-802cd8f1d397 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:27.421271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f1aaa9-d5ab-4a45-a78f-802cd8f1d397 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:27.433372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 887/1776] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:27 2022] GET /identity/v3/roles => generated 1420 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:27.436609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 890/1777] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:27 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:27.437446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba64d265-9677-4682-9bfa-13bc31f434dd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:27.437652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f1aaa9-d5ab-4a45-a78f-802cd8f1d397 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:27.449858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f1aaa9-d5ab-4a45-a78f-802cd8f1d397 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:27.457278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba64d265-9677-4682-9bfa-13bc31f434dd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:27.459332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe5d72d-3c2a-4de5-8e5a-d8a8e5d97cfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:27.460124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe5d72d-3c2a-4de5-8e5a-d8a8e5d97cfd admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:27.460855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe5d72d-3c2a-4de5-8e5a-d8a8e5d97cfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:27.460855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe5d72d-3c2a-4de5-8e5a-d8a8e5d97cfd admin admin] PATH_INFO: `/v3/projects/06294382b1a84a30aad2382c3a7cfcca/users/7d981ab53c564e4c8ecf3adf3af3acd4/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:27.463401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe5d72d-3c2a-4de5-8e5a-d8a8e5d97cfd admin admin] RBAC: Authorizing `identity:create_grant(project_id=06294382b1a84a30aad2382c3a7cfcca, user_id=7d981ab53c564e4c8ecf3adf3af3acd4, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:27.466291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe5d72d-3c2a-4de5-8e5a-d8a8e5d97cfd admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:27.473648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d2ecef-a1a5-40f2-a99a-2c57ef24ba12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:27.476326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2ecef-a1a5-40f2-a99a-2c57ef24ba12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:27.478648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2ecef-a1a5-40f2-a99a-2c57ef24ba12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:27.479104 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d2ecef-a1a5-40f2-a99a-2c57ef24ba12 admin admin] PATH_INFO: `/v3/projects/09200793b1ea4910b6fb9cbe8dd9574b/users/f29bd238a7354d1cbda9b8d820cac6e4/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:27.494296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 891/1778] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:27 2022] PUT /identity/v3/projects/06294382b1a84a30aad2382c3a7cfcca/users/7d981ab53c564e4c8ecf3adf3af3acd4/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:27.508933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e8e592-efac-49a9-b691-3ec7d92d3886 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:27.509474 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e8e592-efac-49a9-b691-3ec7d92d3886 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:27.509879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e8e592-efac-49a9-b691-3ec7d92d3886 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:27.529695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d2ecef-a1a5-40f2-a99a-2c57ef24ba12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=09200793b1ea4910b6fb9cbe8dd9574b, user_id=f29bd238a7354d1cbda9b8d820cac6e4, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:27.535137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d2ecef-a1a5-40f2-a99a-2c57ef24ba12 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:27.559076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:27.595306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 888/1779] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:27 2022] PUT /identity/v3/projects/09200793b1ea4910b6fb9cbe8dd9574b/users/f29bd238a7354d1cbda9b8d820cac6e4/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:27.596512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d2ecef-a1a5-40f2-a99a-2c57ef24ba12 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:27.617170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-20e8e592-efac-49a9-b691-3ec7d92d3886 None None] MFA Rules not processed for user `7d981ab53c564e4c8ecf3adf3af3acd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:27.621967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-20e8e592-efac-49a9-b691-3ec7d92d3886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:27.813244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d2ecef-a1a5-40f2-a99a-2c57ef24ba12 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:27.827655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa8f388-d673-4c76-9577-5e3073a86ea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:27.828533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa8f388-d673-4c76-9577-5e3073a86ea5 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:27.828917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa8f388-d673-4c76-9577-5e3073a86ea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:27.831227 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa8f388-d673-4c76-9577-5e3073a86ea5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:27.832027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa8f388-d673-4c76-9577-5e3073a86ea5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:27.835685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa8f388-d673-4c76-9577-5e3073a86ea5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:27.846148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 892/1780] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:27 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 339 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:27.874372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e8e592-efac-49a9-b691-3ec7d92d3886 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:27.880745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 889/1781] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:27 2022] GET /identity/v3/roles => generated 1420 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:27.888666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa8f388-d673-4c76-9577-5e3073a86ea5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:27.904793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e8e592-efac-49a9-b691-3ec7d92d3886 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:27.923401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fee532-c938-428b-966c-f2987d5ae264 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:27.924545 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fee532-c938-428b-966c-f2987d5ae264 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:27.925660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fee532-c938-428b-966c-f2987d5ae264 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:27.929251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fee532-c938-428b-966c-f2987d5ae264 admin admin] PATH_INFO: `/v3/projects/09200793b1ea4910b6fb9cbe8dd9574b/users/f29bd238a7354d1cbda9b8d820cac6e4/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:27.937304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fee532-c938-428b-966c-f2987d5ae264 admin admin] RBAC: Authorizing `identity:create_grant(project_id=09200793b1ea4910b6fb9cbe8dd9574b, user_id=f29bd238a7354d1cbda9b8d820cac6e4, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:27.947323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fee532-c938-428b-966c-f2987d5ae264 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:28.003664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 893/1782] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:27 2022] PUT /identity/v3/projects/09200793b1ea4910b6fb9cbe8dd9574b/users/f29bd238a7354d1cbda9b8d820cac6e4/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:28.021581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d6da0f-1a63-41fe-9898-5852ff518fa0 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:28.022242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d6da0f-1a63-41fe-9898-5852ff518fa0 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:28.022741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d6da0f-1a63-41fe-9898-5852ff518fa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:28.084836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:28.128046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa8f388-d673-4c76-9577-5e3073a86ea5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:28.172445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-c7d6da0f-1a63-41fe-9898-5852ff518fa0 None None] MFA Rules not processed for user `f29bd238a7354d1cbda9b8d820cac6e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:28.176412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c7d6da0f-1a63-41fe-9898-5852ff518fa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:28.335259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d227133-9c79-4bdd-a56f-f7fb3407f743 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:28.335944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d227133-9c79-4bdd-a56f-f7fb3407f743 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:28.336230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d227133-9c79-4bdd-a56f-f7fb3407f743 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:28.336530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d227133-9c79-4bdd-a56f-f7fb3407f743 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:28.500133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 894/1783] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:28 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 481 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:28.528502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d227133-9c79-4bdd-a56f-f7fb3407f743 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:28.530097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d227133-9c79-4bdd-a56f-f7fb3407f743 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:28.590213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 890/1784] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:27 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 703 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:28.698122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d6da0f-1a63-41fe-9898-5852ff518fa0 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:28.713812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d6da0f-1a63-41fe-9898-5852ff518fa0 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:28.732207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5217f5-cee5-4615-9579-85d52e378cef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:28.735121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5217f5-cee5-4615-9579-85d52e378cef service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:28.736920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5217f5-cee5-4615-9579-85d52e378cef service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:28.736920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5217f5-cee5-4615-9579-85d52e378cef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:28.807503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5217f5-cee5-4615-9579-85d52e378cef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:28.810374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5217f5-cee5-4615-9579-85d52e378cef service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:28.825451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 895/1785] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:28 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:28.955315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d227133-9c79-4bdd-a56f-f7fb3407f743 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:28.967523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d227133-9c79-4bdd-a56f-f7fb3407f743 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:28.978578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52beacf-fcc6-4574-8097-7665cbce8875 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:28.981323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52beacf-fcc6-4574-8097-7665cbce8875 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:28.983847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52beacf-fcc6-4574-8097-7665cbce8875 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:28.985086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52beacf-fcc6-4574-8097-7665cbce8875 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:29.143773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52beacf-fcc6-4574-8097-7665cbce8875 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:29.145558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52beacf-fcc6-4574-8097-7665cbce8875 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:29.209007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 891/1786] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:28 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:29.226000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5463d661-0c52-4ff1-8e95-91e519940408 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:29.226991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5463d661-0c52-4ff1-8e95-91e519940408 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:29.228017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5463d661-0c52-4ff1-8e95-91e519940408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:29.327267 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-5463d661-0c52-4ff1-8e95-91e519940408 None None] MFA Rules not processed for user `fb54ba1355004e14b8dc43cf3c028e67`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:29.331970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5463d661-0c52-4ff1-8e95-91e519940408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:29.450727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:29.495867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 896/1787] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:29 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:29.591970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52beacf-fcc6-4574-8097-7665cbce8875 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:29.603023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52beacf-fcc6-4574-8097-7665cbce8875 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:29.615794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb8c45a-247d-475a-951b-29423f009499 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:29.616508 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8c45a-247d-475a-951b-29423f009499 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:29.616749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8c45a-247d-475a-951b-29423f009499 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:29.617028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb8c45a-247d-475a-951b-29423f009499 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:29.770657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:29.835656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb8c45a-247d-475a-951b-29423f009499 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:29.838175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb8c45a-247d-475a-951b-29423f009499 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:29.862765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 892/1788] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:29 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:30.100199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab624bee-1924-4144-ae0c-6c95472d4222 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:30.100893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab624bee-1924-4144-ae0c-6c95472d4222 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:30.101327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab624bee-1924-4144-ae0c-6c95472d4222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:30.175365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-ab624bee-1924-4144-ae0c-6c95472d4222 None None] MFA Rules not processed for user `bc62747b8aab40d9bff9d853b9b73199`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:30.181422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ab624bee-1924-4144-ae0c-6c95472d4222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:30.360494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daddc5f3-1fd3-41cb-9e76-0e9aa9fc8fb1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:30.361161 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daddc5f3-1fd3-41cb-9e76-0e9aa9fc8fb1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:30.361766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daddc5f3-1fd3-41cb-9e76-0e9aa9fc8fb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:30.481078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-daddc5f3-1fd3-41cb-9e76-0e9aa9fc8fb1 None None] MFA Rules not processed for user `06043e862a394764a58003f205cbb484`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:30.490466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-daddc5f3-1fd3-41cb-9e76-0e9aa9fc8fb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:30.534980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 897/1789] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:30 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 436 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:30.537761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab624bee-1924-4144-ae0c-6c95472d4222 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:30.644967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 893/1790] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:30 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:30.693161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab624bee-1924-4144-ae0c-6c95472d4222 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:30.699216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:30.700040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:30.700463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:30.700888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] PATH_INFO: `/v3/users/59aaa75b0a324985aca9cb823ce1188c` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:30.703186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] RBAC: Authorizing `identity:delete_user(user_id=59aaa75b0a324985aca9cb823ce1188c)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:30.705330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:30.830870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59aaa75b0a324985aca9cb823ce1188c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:30.850025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59aaa75b0a324985aca9cb823ce1188c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:30.851001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59aaa75b0a324985aca9cb823ce1188c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:30.852061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59aaa75b0a324985aca9cb823ce1188c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:30.912252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59aaa75b0a324985aca9cb823ce1188c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:30.970619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 898/1791] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:30 2022] DELETE /identity/v3/users/59aaa75b0a324985aca9cb823ce1188c => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:30.972640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb35583-3a18-42dc-9800-e5140b3c33f6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:30.973936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb35583-3a18-42dc-9800-e5140b3c33f6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:30.974499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb35583-3a18-42dc-9800-e5140b3c33f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:31.015046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5cb35583-3a18-42dc-9800-e5140b3c33f6 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:31.017641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5cb35583-3a18-42dc-9800-e5140b3c33f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:31.198578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 894/1792] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:31.271942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:31.274688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:31.331399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:31.465161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d3726b-6ad2-453d-920a-c1d46e83d591 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:31.475736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830d974e-4b6f-4b2e-b057-82daaeb816c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:31.476654 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830d974e-4b6f-4b2e-b057-82daaeb816c5 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:31.477995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830d974e-4b6f-4b2e-b057-82daaeb816c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:31.478735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830d974e-4b6f-4b2e-b057-82daaeb816c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:31.520291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830d974e-4b6f-4b2e-b057-82daaeb816c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:31.521931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830d974e-4b6f-4b2e-b057-82daaeb816c5 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:31.561556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 899/1793] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:31.874850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaf504c-84ba-483e-803b-97bedc4ce379 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:31.875373 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaf504c-84ba-483e-803b-97bedc4ce379 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:31.875373 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaf504c-84ba-483e-803b-97bedc4ce379 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:31.986653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-dfaf504c-84ba-483e-803b-97bedc4ce379 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:31.988069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-dfaf504c-84ba-483e-803b-97bedc4ce379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:32.156501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:32.183467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 895/1794] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:36:31 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:32.441706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830d974e-4b6f-4b2e-b057-82daaeb816c5 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:32.443679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-830d974e-4b6f-4b2e-b057-82daaeb816c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:32.771934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830d974e-4b6f-4b2e-b057-82daaeb816c5 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:32.803540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a3c485-c70a-4e59-9828-0d73b6bbe9f0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:32.804779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a3c485-c70a-4e59-9828-0d73b6bbe9f0 service swift] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:32.805373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a3c485-c70a-4e59-9828-0d73b6bbe9f0 service swift] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:32.805855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a3c485-c70a-4e59-9828-0d73b6bbe9f0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:32.819119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a3c485-c70a-4e59-9828-0d73b6bbe9f0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:32.821583 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a3c485-c70a-4e59-9828-0d73b6bbe9f0 service swift] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:32.870927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 900/1795] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:36:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:36:33.237472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaf504c-84ba-483e-803b-97bedc4ce379 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:33.245570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-dfaf504c-84ba-483e-803b-97bedc4ce379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:33.295325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaf504c-84ba-483e-803b-97bedc4ce379 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:33.349387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:33.350566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:33.352893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:33.355692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] PATH_INFO: `/v3/projects/3fe4653e43a14bb7b7bbc2f6cbd008cb` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:33.361951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3fe4653e43a14bb7b7bbc2f6cbd008cb)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:33.370492 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:33.628903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3fe4653e43a14bb7b7bbc2f6cbd008cb', 'request_id': 'req-3a831056-5a96-4c08-a6bc-37a4843a3db7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:33.685764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3fe4653e43a14bb7b7bbc2f6cbd008cb', 'request_id': 'req-3a831056-5a96-4c08-a6bc-37a4843a3db7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:33.719100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3fe4653e43a14bb7b7bbc2f6cbd008cb', 'request_id': 'req-3a831056-5a96-4c08-a6bc-37a4843a3db7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:33.743875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] The token cache is being invalidate because project 3fe4653e43a14bb7b7bbc2f6cbd008cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:33.744777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:33.748703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:33.752076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 896/1796] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:33 2022] DELETE /identity/v3/projects/3fe4653e43a14bb7b7bbc2f6cbd008cb => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:33.756641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a3c485-c70a-4e59-9828-0d73b6bbe9f0 service swift] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:33.786903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a3c485-c70a-4e59-9828-0d73b6bbe9f0 service swift] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:34.034241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:34.035655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:34.036124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:34.036569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] PATH_INFO: `/v3/users/7d981ab53c564e4c8ecf3adf3af3acd4` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:34.041817 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d981ab53c564e4c8ecf3adf3af3acd4)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:34.048390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:34.143730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:34.168747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d981ab53c564e4c8ecf3adf3af3acd4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:34.185698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d981ab53c564e4c8ecf3adf3af3acd4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:34.186532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d981ab53c564e4c8ecf3adf3af3acd4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:34.187263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d981ab53c564e4c8ecf3adf3af3acd4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:34.284170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-55ecbdc3-96b7-44a4-b651-bc47034f72f7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d981ab53c564e4c8ecf3adf3af3acd4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:34.371746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 901/1797] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:33 2022] DELETE /identity/v3/users/7d981ab53c564e4c8ecf3adf3af3acd4 => generated 0 bytes in 616 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:34.916906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:34.918482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:34.946379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a831056-5a96-4c08-a6bc-37a4843a3db7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:35.126852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:35.127507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:35.127793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:35.128072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] PATH_INFO: `/v3/projects/06294382b1a84a30aad2382c3a7cfcca` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:35.130482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=06294382b1a84a30aad2382c3a7cfcca)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:35.139377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:35.377294 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '06294382b1a84a30aad2382c3a7cfcca', 'request_id': 'req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:35.410924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '06294382b1a84a30aad2382c3a7cfcca', 'request_id': 'req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:35.436937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '06294382b1a84a30aad2382c3a7cfcca', 'request_id': 'req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:35.507358 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] The token cache is being invalidate because project 06294382b1a84a30aad2382c3a7cfcca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:35.507648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:35.513686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63336cd3-a55a-43c2-9f7a-9a8e26ca5b11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:35.515762 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 897/1798] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:34 2022] DELETE /identity/v3/projects/06294382b1a84a30aad2382c3a7cfcca => generated 0 bytes in 599 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:35.523213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec6c56c-d194-49f8-986c-c93cbc028ee1 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:35.523704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec6c56c-d194-49f8-986c-c93cbc028ee1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:35.523912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec6c56c-d194-49f8-986c-c93cbc028ee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:35.616256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-bec6c56c-d194-49f8-986c-c93cbc028ee1 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:35.617736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-bec6c56c-d194-49f8-986c-c93cbc028ee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:35.830346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 902/1799] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:35.838686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a962b8-e007-426a-a35b-80741e6f1b4d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:35.839318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a962b8-e007-426a-a35b-80741e6f1b4d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:35.839816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a962b8-e007-426a-a35b-80741e6f1b4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:35.877346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f9a962b8-e007-426a-a35b-80741e6f1b4d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:35.880818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f9a962b8-e007-426a-a35b-80741e6f1b4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:35.896866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 898/1800] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:35.900533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec6c56c-d194-49f8-986c-c93cbc028ee1 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:35.914454 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec6c56c-d194-49f8-986c-c93cbc028ee1 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:35.923293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91c44e3-6a83-4e27-88f3-145e2b4fca36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:35.924330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91c44e3-6a83-4e27-88f3-145e2b4fca36 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:35.924897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91c44e3-6a83-4e27-88f3-145e2b4fca36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:35.925349 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91c44e3-6a83-4e27-88f3-145e2b4fca36 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:35.927344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91c44e3-6a83-4e27-88f3-145e2b4fca36 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:35.927770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91c44e3-6a83-4e27-88f3-145e2b4fca36 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:35.930441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91c44e3-6a83-4e27-88f3-145e2b4fca36 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:35.955021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 903/1801] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:35.956366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a962b8-e007-426a-a35b-80741e6f1b4d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:35.962679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:35.978929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a962b8-e007-426a-a35b-80741e6f1b4d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:35.987821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc66fa3b-761b-4761-b62f-bd7553c10bf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:35.988794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc66fa3b-761b-4761-b62f-bd7553c10bf4 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:35.990747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc66fa3b-761b-4761-b62f-bd7553c10bf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:35.991575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc66fa3b-761b-4761-b62f-bd7553c10bf4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:35.997784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc66fa3b-761b-4761-b62f-bd7553c10bf4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:36.001629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc66fa3b-761b-4761-b62f-bd7553c10bf4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:36.061485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 899/1802] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:35 2022] POST /identity/v3/projects => generated 407 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:36.066873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91c44e3-6a83-4e27-88f3-145e2b4fca36 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:36.098467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91c44e3-6a83-4e27-88f3-145e2b4fca36 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:36.201623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4df7d0-2232-4e69-a0d1-f3a71d89417b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:36.202445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4df7d0-2232-4e69-a0d1-f3a71d89417b admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:36.202718 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4df7d0-2232-4e69-a0d1-f3a71d89417b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:36.203495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4df7d0-2232-4e69-a0d1-f3a71d89417b admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:36.204147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4df7d0-2232-4e69-a0d1-f3a71d89417b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:36.206759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4df7d0-2232-4e69-a0d1-f3a71d89417b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:36.249919 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 904/1803] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:36 2022] POST /identity/v3/users => generated 304 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:36:36.254892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc66fa3b-761b-4761-b62f-bd7553c10bf4 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:36.275355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc66fa3b-761b-4761-b62f-bd7553c10bf4 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:36.283218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa739cc-82c6-42f5-8cbd-0f9ed0ca8bea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:36.283988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa739cc-82c6-42f5-8cbd-0f9ed0ca8bea admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:36.284535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa739cc-82c6-42f5-8cbd-0f9ed0ca8bea admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:36.285059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa739cc-82c6-42f5-8cbd-0f9ed0ca8bea admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:36.286142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa739cc-82c6-42f5-8cbd-0f9ed0ca8bea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:36.288876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa739cc-82c6-42f5-8cbd-0f9ed0ca8bea admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:36.306438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 900/1804] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:36 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:36.310713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4df7d0-2232-4e69-a0d1-f3a71d89417b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:36.329691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4df7d0-2232-4e69-a0d1-f3a71d89417b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:36.337430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e494edae-6cca-4897-8998-a832118e20e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:36.339792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e494edae-6cca-4897-8998-a832118e20e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:36.340226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e494edae-6cca-4897-8998-a832118e20e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:36.341203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e494edae-6cca-4897-8998-a832118e20e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:36.341203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e494edae-6cca-4897-8998-a832118e20e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:36.343379 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e494edae-6cca-4897-8998-a832118e20e8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:36.362722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa739cc-82c6-42f5-8cbd-0f9ed0ca8bea admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:36.364544 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 905/1805] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:36 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:36.374670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa739cc-82c6-42f5-8cbd-0f9ed0ca8bea admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:36.384109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e86bd3-6692-4e36-aaa4-c5f72ab9b9b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:36.385908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e86bd3-6692-4e36-aaa4-c5f72ab9b9b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:36.386186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e86bd3-6692-4e36-aaa4-c5f72ab9b9b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:36.386463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e86bd3-6692-4e36-aaa4-c5f72ab9b9b4 admin admin] PATH_INFO: `/v3/projects/9a2313534557434d8e90ae45f2ed7d9b/users/779eabdd826e499381d890564378b93b/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:36.451552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e86bd3-6692-4e36-aaa4-c5f72ab9b9b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a2313534557434d8e90ae45f2ed7d9b, user_id=779eabdd826e499381d890564378b93b, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:36.455063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e86bd3-6692-4e36-aaa4-c5f72ab9b9b4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:36.493146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 901/1806] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:36 2022] PUT /identity/v3/projects/9a2313534557434d8e90ae45f2ed7d9b/users/779eabdd826e499381d890564378b93b/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:36.503903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2ccc1c-9349-42c1-b4a0-db44b55895d1 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:36.504287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2ccc1c-9349-42c1-b4a0-db44b55895d1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:36.507868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2ccc1c-9349-42c1-b4a0-db44b55895d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:36.613237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-ff2ccc1c-9349-42c1-b4a0-db44b55895d1 None None] MFA Rules not processed for user `779eabdd826e499381d890564378b93b`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:36.615403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ff2ccc1c-9349-42c1-b4a0-db44b55895d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:36.624142 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:36.737121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 906/1807] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:36 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:36.742065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e86bd3-6692-4e36-aaa4-c5f72ab9b9b4 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:36.935661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e86bd3-6692-4e36-aaa4-c5f72ab9b9b4 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:36.952116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6cee98-75a0-447a-8886-f0a532704781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:36.952769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6cee98-75a0-447a-8886-f0a532704781 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:36.953053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6cee98-75a0-447a-8886-f0a532704781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:36.953328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6cee98-75a0-447a-8886-f0a532704781 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:36.953970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6cee98-75a0-447a-8886-f0a532704781 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:36.955894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6cee98-75a0-447a-8886-f0a532704781 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:36.962136 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08414c86-185a-4d97-a794-07a20918acae None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:36.962639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08414c86-185a-4d97-a794-07a20918acae None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:36.963071 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08414c86-185a-4d97-a794-07a20918acae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:37.022674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 902/1808] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:36 2022] POST /identity/v3/projects => generated 409 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:37.027171 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6cee98-75a0-447a-8886-f0a532704781 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:37.036178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-08414c86-185a-4d97-a794-07a20918acae None None] MFA Rules not processed for user `0478ebab37fb4f488eef84f7a241e206`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:37.039406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-08414c86-185a-4d97-a794-07a20918acae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:37.247548 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 907/1809] 10.222.0.52 () {58 vars in 1015 bytes} [Mon Sep 26 01:36:36 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:37.266234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6cee98-75a0-447a-8886-f0a532704781 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:37.283405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad64ad64-6d7e-4ad8-a1c3-becbb4e9581d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:37.284494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad64ad64-6d7e-4ad8-a1c3-becbb4e9581d admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:37.284814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad64ad64-6d7e-4ad8-a1c3-becbb4e9581d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:37.284996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad64ad64-6d7e-4ad8-a1c3-becbb4e9581d admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:37.285757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad64ad64-6d7e-4ad8-a1c3-becbb4e9581d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:37.294031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad64ad64-6d7e-4ad8-a1c3-becbb4e9581d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:37.330424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 903/1810] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:37 2022] POST /identity/v3/users => generated 311 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:36:37.331434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08414c86-185a-4d97-a794-07a20918acae None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:37.335631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-08414c86-185a-4d97-a794-07a20918acae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:37.337640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad64ad64-6d7e-4ad8-a1c3-becbb4e9581d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:37.351031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08414c86-185a-4d97-a794-07a20918acae None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:37.363394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d082e4-d8f8-423f-ba1e-17e3a2c82227 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:37.364055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d082e4-d8f8-423f-ba1e-17e3a2c82227 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:37.364362 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d082e4-d8f8-423f-ba1e-17e3a2c82227 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:37.364652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d082e4-d8f8-423f-ba1e-17e3a2c82227 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:37.366258 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad64ad64-6d7e-4ad8-a1c3-becbb4e9581d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:37.391935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f312845d-9470-478d-b638-ab5178f22784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:37.392739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f312845d-9470-478d-b638-ab5178f22784 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:37.392876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f312845d-9470-478d-b638-ab5178f22784 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:37.393163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f312845d-9470-478d-b638-ab5178f22784 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:37.393926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f312845d-9470-478d-b638-ab5178f22784 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:37.395149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d082e4-d8f8-423f-ba1e-17e3a2c82227 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:37.396805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d082e4-d8f8-423f-ba1e-17e3a2c82227 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:37.397545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f312845d-9470-478d-b638-ab5178f22784 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:37.428385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 904/1811] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:37 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:37.428385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f312845d-9470-478d-b638-ab5178f22784 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:37.430927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f312845d-9470-478d-b638-ab5178f22784 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:37.434231 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 908/1812] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:36:37.435085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d082e4-d8f8-423f-ba1e-17e3a2c82227 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:37.452977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f312845d-9470-478d-b638-ab5178f22784 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:37.457308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fda6d2-0536-4e37-b696-5e927bf32714 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:37.458162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fda6d2-0536-4e37-b696-5e927bf32714 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:37.461886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fda6d2-0536-4e37-b696-5e927bf32714 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:37.462415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fda6d2-0536-4e37-b696-5e927bf32714 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:37.478131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d082e4-d8f8-423f-ba1e-17e3a2c82227 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:37.481574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:37.488291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fda6d2-0536-4e37-b696-5e927bf32714 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:37.490491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fda6d2-0536-4e37-b696-5e927bf32714 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:37.508114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad125786-41b7-4589-b74d-6d49f4d523b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:37.519202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 905/1813] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:37.540691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad125786-41b7-4589-b74d-6d49f4d523b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:37.540691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad125786-41b7-4589-b74d-6d49f4d523b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:37.540691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad125786-41b7-4589-b74d-6d49f4d523b5 admin admin] PATH_INFO: `/v3/projects/e0c748b360d94e3ca417ed4eb257eb80/users/c23880d8453c4acb9c8b9278db573799/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:37.585319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad125786-41b7-4589-b74d-6d49f4d523b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0c748b360d94e3ca417ed4eb257eb80, user_id=c23880d8453c4acb9c8b9278db573799, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:37.593856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad125786-41b7-4589-b74d-6d49f4d523b5 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:37.631683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 909/1814] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:37 2022] PUT /identity/v3/projects/e0c748b360d94e3ca417ed4eb257eb80/users/c23880d8453c4acb9c8b9278db573799/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:37.632624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fda6d2-0536-4e37-b696-5e927bf32714 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:37.664540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fda6d2-0536-4e37-b696-5e927bf32714 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:37.937138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdf2fec-9dda-413e-860e-5c56f67d009d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:37.944102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdf2fec-9dda-413e-860e-5c56f67d009d admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:37.944590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdf2fec-9dda-413e-860e-5c56f67d009d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:37.946435 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdf2fec-9dda-413e-860e-5c56f67d009d admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:37.954266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdf2fec-9dda-413e-860e-5c56f67d009d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:37.972593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdf2fec-9dda-413e-860e-5c56f67d009d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:38.033076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 906/1815] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:37 2022] GET /identity/v3/roles => generated 1420 bytes in 401 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:38.038453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad125786-41b7-4589-b74d-6d49f4d523b5 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:38.059823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad125786-41b7-4589-b74d-6d49f4d523b5 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:38.094583 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9052367-1b10-4641-9278-fb950a452ea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:38.095570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9052367-1b10-4641-9278-fb950a452ea4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:38.095860 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9052367-1b10-4641-9278-fb950a452ea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:38.097312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9052367-1b10-4641-9278-fb950a452ea4 admin admin] PATH_INFO: `/v3/projects/e0c748b360d94e3ca417ed4eb257eb80/users/c23880d8453c4acb9c8b9278db573799/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:38.100819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9052367-1b10-4641-9278-fb950a452ea4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0c748b360d94e3ca417ed4eb257eb80, user_id=c23880d8453c4acb9c8b9278db573799, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:38.122455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9052367-1b10-4641-9278-fb950a452ea4 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:38.177945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 910/1816] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:38 2022] PUT /identity/v3/projects/e0c748b360d94e3ca417ed4eb257eb80/users/c23880d8453c4acb9c8b9278db573799/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:38.179070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3543fa9-df37-499a-85a1-3000f47299fd None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:38.179469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3543fa9-df37-499a-85a1-3000f47299fd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:38.179758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3543fa9-df37-499a-85a1-3000f47299fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:38.287050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b3543fa9-df37-499a-85a1-3000f47299fd None None] MFA Rules not processed for user `c23880d8453c4acb9c8b9278db573799`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:38.292189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b3543fa9-df37-499a-85a1-3000f47299fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:38.673593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 907/1817] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:38 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 495 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:38.949530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9052367-1b10-4641-9278-fb950a452ea4 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:38.951989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f9052367-1b10-4641-9278-fb950a452ea4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:39.101427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9052367-1b10-4641-9278-fb950a452ea4 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:39.110186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee0aa5a-027d-4e59-bd3b-15ee1df3361e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:39.111022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee0aa5a-027d-4e59-bd3b-15ee1df3361e service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:39.111941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee0aa5a-027d-4e59-bd3b-15ee1df3361e service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:39.112332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee0aa5a-027d-4e59-bd3b-15ee1df3361e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:39.141528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee0aa5a-027d-4e59-bd3b-15ee1df3361e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:39.143220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee0aa5a-027d-4e59-bd3b-15ee1df3361e service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:39.166347 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 911/1818] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:38 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:39.939826 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3543fa9-df37-499a-85a1-3000f47299fd None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:39.946634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b3543fa9-df37-499a-85a1-3000f47299fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:39.995777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3543fa9-df37-499a-85a1-3000f47299fd None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:40.012125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1817bc2c-b463-4eaa-a157-7b88ad9f7b21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:40.013232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1817bc2c-b463-4eaa-a157-7b88ad9f7b21 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:40.013909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1817bc2c-b463-4eaa-a157-7b88ad9f7b21 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:40.014393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1817bc2c-b463-4eaa-a157-7b88ad9f7b21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:40.154308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1817bc2c-b463-4eaa-a157-7b88ad9f7b21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:40.157870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1817bc2c-b463-4eaa-a157-7b88ad9f7b21 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:40.175985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 908/1819] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:39 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:40.672211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51ba9e4-7342-48b6-a7d0-05e20f70e87d None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:40.672824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51ba9e4-7342-48b6-a7d0-05e20f70e87d None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:40.672824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51ba9e4-7342-48b6-a7d0-05e20f70e87d None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:40.681490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 912/1820] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:36:40 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:40.741440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4e4db0-e0a2-4f7c-895d-e3537757b30e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:40.742330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4e4db0-e0a2-4f7c-895d-e3537757b30e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:40.743089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4e4db0-e0a2-4f7c-895d-e3537757b30e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:40.786579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bb4e4db0-e0a2-4f7c-895d-e3537757b30e None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:40.787971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bb4e4db0-e0a2-4f7c-895d-e3537757b30e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:40.816395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:40.931309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 909/1821] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:36:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:41.677770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51ba9e4-7342-48b6-a7d0-05e20f70e87d None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:41.680222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e51ba9e4-7342-48b6-a7d0-05e20f70e87d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:41.753081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51ba9e4-7342-48b6-a7d0-05e20f70e87d None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:41.780525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cbdd27-ff59-433a-90a1-069b19ca238d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:41.781346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cbdd27-ff59-433a-90a1-069b19ca238d service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:41.781785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cbdd27-ff59-433a-90a1-069b19ca238d service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:41.782713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cbdd27-ff59-433a-90a1-069b19ca238d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:41.808210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:41.819273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cbdd27-ff59-433a-90a1-069b19ca238d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:41.822811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cbdd27-ff59-433a-90a1-069b19ca238d service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:41.885800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 913/1822] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:42.144159 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4e4db0-e0a2-4f7c-895d-e3537757b30e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:42.146469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bb4e4db0-e0a2-4f7c-895d-e3537757b30e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:42.303511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4e4db0-e0a2-4f7c-895d-e3537757b30e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:42.313381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:42.314042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:42.314328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:42.315286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] PATH_INFO: `/v3/users/7eea683669f846c08e9f9751270f24fa` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:42.317318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] RBAC: Authorizing `identity:delete_user(user_id=7eea683669f846c08e9f9751270f24fa)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:42.319840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:42.510366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7eea683669f846c08e9f9751270f24fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:42.530477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7eea683669f846c08e9f9751270f24fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:42.531487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7eea683669f846c08e9f9751270f24fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:42.532464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7eea683669f846c08e9f9751270f24fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:42.560823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7eea683669f846c08e9f9751270f24fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:42.613543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 910/1823] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:42 2022] DELETE /identity/v3/users/7eea683669f846c08e9f9751270f24fa => generated 0 bytes in 470 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:42.618342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fa1c12-e56d-4a26-9389-7a029d63ee3f None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:42.618874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fa1c12-e56d-4a26-9389-7a029d63ee3f None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:42.619412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fa1c12-e56d-4a26-9389-7a029d63ee3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:42.657571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-48fa1c12-e56d-4a26-9389-7a029d63ee3f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:42.678761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-48fa1c12-e56d-4a26-9389-7a029d63ee3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:42.885563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 914/1824] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:43.132032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:43.134471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:43.212119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:43.273543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8404224f-7558-47e3-83d7-8e00711cdf2b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:43.287233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1392c2f8-ef91-4f8d-8b12-db6de0ddb306 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:43.287877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1392c2f8-ef91-4f8d-8b12-db6de0ddb306 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:43.288270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1392c2f8-ef91-4f8d-8b12-db6de0ddb306 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:43.288472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1392c2f8-ef91-4f8d-8b12-db6de0ddb306 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:43.315225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1392c2f8-ef91-4f8d-8b12-db6de0ddb306 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:43.316733 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1392c2f8-ef91-4f8d-8b12-db6de0ddb306 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:43.343537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 911/1825] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:43.865780 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f604cbc-a211-4077-85cf-ccc22f00fda7 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:43.866340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f604cbc-a211-4077-85cf-ccc22f00fda7 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:43.866340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f604cbc-a211-4077-85cf-ccc22f00fda7 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:43.868056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 915/1826] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:36:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:43.900870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e8ea9-4d17-48c8-acc8-f5251f922285 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:43.900870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e8ea9-4d17-48c8-acc8-f5251f922285 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:43.900870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e8ea9-4d17-48c8-acc8-f5251f922285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:43.919659 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b18e8ea9-4d17-48c8-acc8-f5251f922285 None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:43.920948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b18e8ea9-4d17-48c8-acc8-f5251f922285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:44.000170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 912/1827] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:36:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:44.165234 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f604cbc-a211-4077-85cf-ccc22f00fda7 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:44.167204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4f604cbc-a211-4077-85cf-ccc22f00fda7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:44.192940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f604cbc-a211-4077-85cf-ccc22f00fda7 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:44.207827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbd0b3e-183a-446c-afe9-4110a64b2098 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:44.208702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbd0b3e-183a-446c-afe9-4110a64b2098 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:44.208979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbd0b3e-183a-446c-afe9-4110a64b2098 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:44.209249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbd0b3e-183a-446c-afe9-4110a64b2098 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:44.221319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbd0b3e-183a-446c-afe9-4110a64b2098 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:44.223253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbd0b3e-183a-446c-afe9-4110a64b2098 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:44.234488 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 916/1828] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:44.330810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18e8ea9-4d17-48c8-acc8-f5251f922285 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:44.332319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b18e8ea9-4d17-48c8-acc8-f5251f922285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:44.353606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18e8ea9-4d17-48c8-acc8-f5251f922285 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:44.360011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:44.360935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:44.361282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:44.361735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] PATH_INFO: `/v3/projects/db5fce1fe9214c48b0aff3c0de81a8b5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:44.362968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db5fce1fe9214c48b0aff3c0de81a8b5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:44.364917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:44.415259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db5fce1fe9214c48b0aff3c0de81a8b5', 'request_id': 'req-b8b9c861-2608-415a-bf85-6257033dddb8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:44.421611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db5fce1fe9214c48b0aff3c0de81a8b5', 'request_id': 'req-b8b9c861-2608-415a-bf85-6257033dddb8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:44.430118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db5fce1fe9214c48b0aff3c0de81a8b5', 'request_id': 'req-b8b9c861-2608-415a-bf85-6257033dddb8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:44.440613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] The token cache is being invalidate because project db5fce1fe9214c48b0aff3c0de81a8b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:44.440909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:44.441556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:44.446717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 913/1829] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:44 2022] DELETE /identity/v3/projects/db5fce1fe9214c48b0aff3c0de81a8b5 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:44.447787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbd0b3e-183a-446c-afe9-4110a64b2098 service nova] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:44.458399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbd0b3e-183a-446c-afe9-4110a64b2098 service nova] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:44.527754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:44.528408 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:44.528854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:44.529292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] PATH_INFO: `/v3/users/f29bd238a7354d1cbda9b8d820cac6e4` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:44.530552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f29bd238a7354d1cbda9b8d820cac6e4)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:44.532676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:44.605200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f29bd238a7354d1cbda9b8d820cac6e4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:44.615345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f29bd238a7354d1cbda9b8d820cac6e4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:44.619211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f29bd238a7354d1cbda9b8d820cac6e4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:44.619211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f29bd238a7354d1cbda9b8d820cac6e4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:44.635376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e09efe76-6647-40b2-a4c6-2174c780f122 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f29bd238a7354d1cbda9b8d820cac6e4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:44.650236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 917/1830] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:44 2022] DELETE /identity/v3/users/f29bd238a7354d1cbda9b8d820cac6e4 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:44.933770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:44.935637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:44.952650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b9c861-2608-415a-bf85-6257033dddb8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:44.999337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:45.017653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.018288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.018591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.018861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] PATH_INFO: `/v3/projects/09200793b1ea4910b6fb9cbe8dd9574b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.019979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=09200793b1ea4910b6fb9cbe8dd9574b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.021945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:45.080834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09200793b1ea4910b6fb9cbe8dd9574b', 'request_id': 'req-6ac31faa-7cc7-4175-96be-27a242ccad8f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:45.088182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09200793b1ea4910b6fb9cbe8dd9574b', 'request_id': 'req-6ac31faa-7cc7-4175-96be-27a242ccad8f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:45.095950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09200793b1ea4910b6fb9cbe8dd9574b', 'request_id': 'req-6ac31faa-7cc7-4175-96be-27a242ccad8f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:45.111092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] The token cache is being invalidate because project 09200793b1ea4910b6fb9cbe8dd9574b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:45.111092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:45.111092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6ac31faa-7cc7-4175-96be-27a242ccad8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:45.112861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 914/1831] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:44 2022] DELETE /identity/v3/projects/09200793b1ea4910b6fb9cbe8dd9574b => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:45.118075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad266c-dda2-49e4-bd96-2d552fdc8c47 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.118548 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad266c-dda2-49e4-bd96-2d552fdc8c47 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.118668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad266c-dda2-49e4-bd96-2d552fdc8c47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.126462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:45.144950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b0ad266c-dda2-49e4-bd96-2d552fdc8c47 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:45.146185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b0ad266c-dda2-49e4-bd96-2d552fdc8c47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:45.221726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 918/1832] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:45.225800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f1ad29-0595-4cc7-8b9e-739f5813ab11 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.226116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f1ad29-0595-4cc7-8b9e-739f5813ab11 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.227229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f1ad29-0595-4cc7-8b9e-739f5813ab11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.249081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-17f1ad29-0595-4cc7-8b9e-739f5813ab11 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:45.250324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-17f1ad29-0595-4cc7-8b9e-739f5813ab11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:45.260400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 915/1833] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:45.265397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ad266c-dda2-49e4-bd96-2d552fdc8c47 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:45.276374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ad266c-dda2-49e4-bd96-2d552fdc8c47 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:45.282095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0e7d0d-d15d-45df-8907-ffa1ab9423b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.283064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0e7d0d-d15d-45df-8907-ffa1ab9423b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.283414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0e7d0d-d15d-45df-8907-ffa1ab9423b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.283783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0e7d0d-d15d-45df-8907-ffa1ab9423b2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.285192 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0e7d0d-d15d-45df-8907-ffa1ab9423b2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:45.285578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0e7d0d-d15d-45df-8907-ffa1ab9423b2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.287610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0e7d0d-d15d-45df-8907-ffa1ab9423b2 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:45.306839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 919/1834] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:45.308024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f1ad29-0595-4cc7-8b9e-739f5813ab11 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:45.318766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f1ad29-0595-4cc7-8b9e-739f5813ab11 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:45.329467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442e3f14-2d75-47fa-bcfb-5c97b2920bd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.330081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e3f14-2d75-47fa-bcfb-5c97b2920bd1 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.330363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e3f14-2d75-47fa-bcfb-5c97b2920bd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.330637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e3f14-2d75-47fa-bcfb-5c97b2920bd1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.331509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442e3f14-2d75-47fa-bcfb-5c97b2920bd1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.333320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442e3f14-2d75-47fa-bcfb-5c97b2920bd1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:45.361806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 916/1835] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:45 2022] POST /identity/v3/projects => generated 369 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:36:45.366060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0e7d0d-d15d-45df-8907-ffa1ab9423b2 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:45.377308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0e7d0d-d15d-45df-8907-ffa1ab9423b2 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:45.437510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1b69c5-d4d8-4028-89da-5907591ddbd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.438041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1b69c5-d4d8-4028-89da-5907591ddbd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.438320 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1b69c5-d4d8-4028-89da-5907591ddbd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.438589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1b69c5-d4d8-4028-89da-5907591ddbd7 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.439230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1b69c5-d4d8-4028-89da-5907591ddbd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.441193 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1b69c5-d4d8-4028-89da-5907591ddbd7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:45.467341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 920/1836] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:45 2022] POST /identity/v3/users => generated 285 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:36:45.468602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442e3f14-2d75-47fa-bcfb-5c97b2920bd1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:45.479574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442e3f14-2d75-47fa-bcfb-5c97b2920bd1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:45.484615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3319e3d9-b871-4586-9a2f-6759076d9b98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.485244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319e3d9-b871-4586-9a2f-6759076d9b98 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.485527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319e3d9-b871-4586-9a2f-6759076d9b98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.485806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319e3d9-b871-4586-9a2f-6759076d9b98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.486470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3319e3d9-b871-4586-9a2f-6759076d9b98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.488120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3319e3d9-b871-4586-9a2f-6759076d9b98 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:45.499952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 917/1837] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:45.503888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1b69c5-d4d8-4028-89da-5907591ddbd7 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:45.515240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1b69c5-d4d8-4028-89da-5907591ddbd7 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:45.519956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed16c5a-a27c-46f1-a41b-ebad81bef311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.520605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed16c5a-a27c-46f1-a41b-ebad81bef311 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.520888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed16c5a-a27c-46f1-a41b-ebad81bef311 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.521162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed16c5a-a27c-46f1-a41b-ebad81bef311 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.521814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed16c5a-a27c-46f1-a41b-ebad81bef311 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.523482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed16c5a-a27c-46f1-a41b-ebad81bef311 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:45.534778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 921/1838] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:45.539053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3319e3d9-b871-4586-9a2f-6759076d9b98 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:45.549656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3319e3d9-b871-4586-9a2f-6759076d9b98 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:45.553821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8625ab3-bf93-4622-9833-f08ef772c5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.555004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8625ab3-bf93-4622-9833-f08ef772c5e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.555414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8625ab3-bf93-4622-9833-f08ef772c5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.555717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8625ab3-bf93-4622-9833-f08ef772c5e7 admin admin] PATH_INFO: `/v3/projects/63c6c2d404074381ac040ad61b306499/users/1b48b6e3926640b791d6dabdacc63299/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.572403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8625ab3-bf93-4622-9833-f08ef772c5e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63c6c2d404074381ac040ad61b306499, user_id=1b48b6e3926640b791d6dabdacc63299, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.575744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8625ab3-bf93-4622-9833-f08ef772c5e7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:45.588699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 918/1839] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:45 2022] PUT /identity/v3/projects/63c6c2d404074381ac040ad61b306499/users/1b48b6e3926640b791d6dabdacc63299/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:45.599179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a43bf4-066a-47bb-83f3-84705a5034c1 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.599489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a43bf4-066a-47bb-83f3-84705a5034c1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.599762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a43bf4-066a-47bb-83f3-84705a5034c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.628749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-95a43bf4-066a-47bb-83f3-84705a5034c1 None None] MFA Rules not processed for user `1b48b6e3926640b791d6dabdacc63299`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:45.630024 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-95a43bf4-066a-47bb-83f3-84705a5034c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:45.689746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:45.699897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 922/1840] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:45 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:45.700833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8625ab3-bf93-4622-9833-f08ef772c5e7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:45.773542 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8625ab3-bf93-4622-9833-f08ef772c5e7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:45.780156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ea848-ed57-4607-84d9-391c43d8bd34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.781413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ea848-ed57-4607-84d9-391c43d8bd34 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.781756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ea848-ed57-4607-84d9-391c43d8bd34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.782024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4ea848-ed57-4607-84d9-391c43d8bd34 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.782742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4ea848-ed57-4607-84d9-391c43d8bd34 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.785353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4ea848-ed57-4607-84d9-391c43d8bd34 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:45.804121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:45.818532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 919/1841] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:45 2022] POST /identity/v3/projects => generated 365 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:36:45.822374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a43bf4-066a-47bb-83f3-84705a5034c1 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:45.833296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a43bf4-066a-47bb-83f3-84705a5034c1 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:45.891229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db703407-0f8e-4066-9666-cb3a1a8a1673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.891840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db703407-0f8e-4066-9666-cb3a1a8a1673 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.892117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db703407-0f8e-4066-9666-cb3a1a8a1673 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.892418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db703407-0f8e-4066-9666-cb3a1a8a1673 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.893044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db703407-0f8e-4066-9666-cb3a1a8a1673 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.894979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db703407-0f8e-4066-9666-cb3a1a8a1673 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:45.917592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 923/1842] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:45 2022] POST /identity/v3/users => generated 289 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:45.921328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ea848-ed57-4607-84d9-391c43d8bd34 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:45.933478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4ea848-ed57-4607-84d9-391c43d8bd34 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:45.938085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a569f7-054e-4752-97be-f7a1ecf04d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.938683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a569f7-054e-4752-97be-f7a1ecf04d1e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.938963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a569f7-054e-4752-97be-f7a1ecf04d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.939265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a569f7-054e-4752-97be-f7a1ecf04d1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.939959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a569f7-054e-4752-97be-f7a1ecf04d1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.941539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a569f7-054e-4752-97be-f7a1ecf04d1e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:45.952441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 920/1843] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:45.956558 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db703407-0f8e-4066-9666-cb3a1a8a1673 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:45.966973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db703407-0f8e-4066-9666-cb3a1a8a1673 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:45.971473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf05a13-bdf2-49ce-a8a6-119326482dfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:45.972112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf05a13-bdf2-49ce-a8a6-119326482dfc admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:45.972415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf05a13-bdf2-49ce-a8a6-119326482dfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:45.972689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf05a13-bdf2-49ce-a8a6-119326482dfc admin admin] PATH_INFO: `/v3/projects/a6cc1328075a48a7a4786f73aad6ae14/users/34c847469bd04e8e9f51546edd2f29f5/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:45.988000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf05a13-bdf2-49ce-a8a6-119326482dfc admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6cc1328075a48a7a4786f73aad6ae14, user_id=34c847469bd04e8e9f51546edd2f29f5, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:45.991413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf05a13-bdf2-49ce-a8a6-119326482dfc admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:46.005624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 924/1844] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:45 2022] PUT /identity/v3/projects/a6cc1328075a48a7a4786f73aad6ae14/users/34c847469bd04e8e9f51546edd2f29f5/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:46.010344 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a569f7-054e-4752-97be-f7a1ecf04d1e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:46.022012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a569f7-054e-4752-97be-f7a1ecf04d1e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:46.115327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c3b9a9-b727-4d99-b3d0-df18cd9b24d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:46.115945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c3b9a9-b727-4d99-b3d0-df18cd9b24d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:46.116224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c3b9a9-b727-4d99-b3d0-df18cd9b24d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:46.116531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c3b9a9-b727-4d99-b3d0-df18cd9b24d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:46.117417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c3b9a9-b727-4d99-b3d0-df18cd9b24d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:46.119060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c3b9a9-b727-4d99-b3d0-df18cd9b24d7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:46.133902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 921/1845] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:46 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:46.135184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf05a13-bdf2-49ce-a8a6-119326482dfc admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:46.146811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf05a13-bdf2-49ce-a8a6-119326482dfc admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:46.152610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6251ba-f7b5-4f51-9c58-002cb27c723d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:46.153282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6251ba-f7b5-4f51-9c58-002cb27c723d admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:46.153559 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6251ba-f7b5-4f51-9c58-002cb27c723d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:46.153830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6251ba-f7b5-4f51-9c58-002cb27c723d admin admin] PATH_INFO: `/v3/projects/a6cc1328075a48a7a4786f73aad6ae14/users/34c847469bd04e8e9f51546edd2f29f5/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:46.155762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6251ba-f7b5-4f51-9c58-002cb27c723d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6cc1328075a48a7a4786f73aad6ae14, user_id=34c847469bd04e8e9f51546edd2f29f5, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:46.158471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6251ba-f7b5-4f51-9c58-002cb27c723d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:46.170497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 925/1846] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:46 2022] PUT /identity/v3/projects/a6cc1328075a48a7a4786f73aad6ae14/users/34c847469bd04e8e9f51546edd2f29f5/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:36:46.182088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d981c-74f5-45ff-adf9-093b622c9530 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:46.182409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d981c-74f5-45ff-adf9-093b622c9530 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:46.182748 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d981c-74f5-45ff-adf9-093b622c9530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:46.221309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fb0d981c-74f5-45ff-adf9-093b622c9530 None None] MFA Rules not processed for user `34c847469bd04e8e9f51546edd2f29f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:46.222765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fb0d981c-74f5-45ff-adf9-093b622c9530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:46.356676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 922/1847] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:46 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:46.369290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6251ba-f7b5-4f51-9c58-002cb27c723d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:46.376128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8b6251ba-f7b5-4f51-9c58-002cb27c723d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:46.497346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6251ba-f7b5-4f51-9c58-002cb27c723d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:46.511923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e2e858-d761-42e8-9ac3-a081a94a6c81 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:46.512739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e2e858-d761-42e8-9ac3-a081a94a6c81 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:46.513091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e2e858-d761-42e8-9ac3-a081a94a6c81 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:46.513537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e2e858-d761-42e8-9ac3-a081a94a6c81 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:46.524994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e2e858-d761-42e8-9ac3-a081a94a6c81 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:46.528975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e2e858-d761-42e8-9ac3-a081a94a6c81 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:46.546765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 926/1848] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:46 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:46.617111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0d981c-74f5-45ff-adf9-093b622c9530 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:46.618727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fb0d981c-74f5-45ff-adf9-093b622c9530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:46.638802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0d981c-74f5-45ff-adf9-093b622c9530 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:46.645688 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8844af47-2f79-447b-8f54-8c1447f0978c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:46.646358 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8844af47-2f79-447b-8f54-8c1447f0978c service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:46.646643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8844af47-2f79-447b-8f54-8c1447f0978c service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:46.646918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8844af47-2f79-447b-8f54-8c1447f0978c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:46.713775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8844af47-2f79-447b-8f54-8c1447f0978c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:46.715234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8844af47-2f79-447b-8f54-8c1447f0978c service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:46.729470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 923/1849] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:46 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:50.474192 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e2e858-d761-42e8-9ac3-a081a94a6c81 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:50.528714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:50.541145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e2e858-d761-42e8-9ac3-a081a94a6c81 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:50.545709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:50.546339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:50.546620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:50.546900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] PATH_INFO: `/v3/users/1b48b6e3926640b791d6dabdacc63299` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:50.548020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b48b6e3926640b791d6dabdacc63299)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:50.549988 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:50.595825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b48b6e3926640b791d6dabdacc63299'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:50.603604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b48b6e3926640b791d6dabdacc63299'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:50.604396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b48b6e3926640b791d6dabdacc63299'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:50.605034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b48b6e3926640b791d6dabdacc63299'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:50.617927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b48b6e3926640b791d6dabdacc63299'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:50.630478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 927/1850] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:50 2022] DELETE /identity/v3/users/1b48b6e3926640b791d6dabdacc63299 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:50.635893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315ef44a-fa77-4f37-9896-d54ebc130ad9 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:50.636436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315ef44a-fa77-4f37-9896-d54ebc130ad9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:50.636603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315ef44a-fa77-4f37-9896-d54ebc130ad9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:50.644915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:50.656183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-315ef44a-fa77-4f37-9896-d54ebc130ad9 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:50.657356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-315ef44a-fa77-4f37-9896-d54ebc130ad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:50.716178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 924/1851] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:50.726188 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:50.727769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:50.788034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1a0989-7f5e-4f3f-a4f6-c952566fe441 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:50.792269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1757f5-4d2d-4404-83fe-20f8dd72babf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:50.793148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1757f5-4d2d-4404-83fe-20f8dd72babf service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:50.793571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1757f5-4d2d-4404-83fe-20f8dd72babf service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:50.793972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1757f5-4d2d-4404-83fe-20f8dd72babf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:50.805965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1757f5-4d2d-4404-83fe-20f8dd72babf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:50.807717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1757f5-4d2d-4404-83fe-20f8dd72babf service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:50.819265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 928/1852] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:51.142890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315ef44a-fa77-4f37-9896-d54ebc130ad9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:51.144740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-315ef44a-fa77-4f37-9896-d54ebc130ad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:51.158720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315ef44a-fa77-4f37-9896-d54ebc130ad9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:51.169068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:51.170937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:51.171356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:51.171740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] PATH_INFO: `/v3/projects/63c6c2d404074381ac040ad61b306499` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:51.173073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=63c6c2d404074381ac040ad61b306499)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:51.184009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:51.232574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63c6c2d404074381ac040ad61b306499', 'request_id': 'req-66f9fa5e-5922-444e-ab14-5660b76631a5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:51.239051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63c6c2d404074381ac040ad61b306499', 'request_id': 'req-66f9fa5e-5922-444e-ab14-5660b76631a5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:51.246039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63c6c2d404074381ac040ad61b306499', 'request_id': 'req-66f9fa5e-5922-444e-ab14-5660b76631a5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:51.256364 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] The token cache is being invalidate because project 63c6c2d404074381ac040ad61b306499 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:51.256678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:51.257400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:51.259168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 925/1853] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:51 2022] DELETE /identity/v3/projects/63c6c2d404074381ac040ad61b306499 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:51.263097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1757f5-4d2d-4404-83fe-20f8dd72babf service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:51.272681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1757f5-4d2d-4404-83fe-20f8dd72babf service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:51.332747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:51.333375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:51.333652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:51.333922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] PATH_INFO: `/v3/users/34c847469bd04e8e9f51546edd2f29f5` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:51.335037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34c847469bd04e8e9f51546edd2f29f5)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:51.337714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:51.354753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:51.387553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34c847469bd04e8e9f51546edd2f29f5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:51.395557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34c847469bd04e8e9f51546edd2f29f5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:51.396224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34c847469bd04e8e9f51546edd2f29f5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:51.396918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34c847469bd04e8e9f51546edd2f29f5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:51.410101 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-86057f5b-fa8a-4713-8f01-66e1c12126a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34c847469bd04e8e9f51546edd2f29f5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:51.424836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 929/1854] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:51 2022] DELETE /identity/v3/users/34c847469bd04e8e9f51546edd2f29f5 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:51.978760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:51.980547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:51.998128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f9fa5e-5922-444e-ab14-5660b76631a5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:52.094372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:52.095479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:52.096848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:52.096848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] PATH_INFO: `/v3/projects/a6cc1328075a48a7a4786f73aad6ae14` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:52.098044 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6cc1328075a48a7a4786f73aad6ae14)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:52.100820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:52.147642 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:52.167309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6cc1328075a48a7a4786f73aad6ae14', 'request_id': 'req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:52.177886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6cc1328075a48a7a4786f73aad6ae14', 'request_id': 'req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:52.195768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6cc1328075a48a7a4786f73aad6ae14', 'request_id': 'req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:52.209091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] The token cache is being invalidate because project a6cc1328075a48a7a4786f73aad6ae14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:52.209466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:52.211118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2fd43cf4-e60f-4ab1-a556-5ceca56933c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:52.213726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 926/1855] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:51 2022] DELETE /identity/v3/projects/a6cc1328075a48a7a4786f73aad6ae14 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:52.218363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0c22eb-8b25-4a8a-ad6c-e86fd39335f5 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:52.218837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0c22eb-8b25-4a8a-ad6c-e86fd39335f5 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:52.219036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0c22eb-8b25-4a8a-ad6c-e86fd39335f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:52.256554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-ec0c22eb-8b25-4a8a-ad6c-e86fd39335f5 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:52.258763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ec0c22eb-8b25-4a8a-ad6c-e86fd39335f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:52.392478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 930/1856] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:52.394658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec8090b-c228-4bfd-831f-442e1a0a07df None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:52.394950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec8090b-c228-4bfd-831f-442e1a0a07df None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:52.395487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec8090b-c228-4bfd-831f-442e1a0a07df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:52.424542 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7ec8090b-c228-4bfd-831f-442e1a0a07df None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:52.425341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7ec8090b-c228-4bfd-831f-442e1a0a07df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:52.434890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 927/1857] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:52.441643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0c22eb-8b25-4a8a-ad6c-e86fd39335f5 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:52.454667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0c22eb-8b25-4a8a-ad6c-e86fd39335f5 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:52.460674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb6f1c6-0510-49e5-b466-d870c98bad0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:52.461706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb6f1c6-0510-49e5-b466-d870c98bad0d admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:52.461998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb6f1c6-0510-49e5-b466-d870c98bad0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:52.462273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb6f1c6-0510-49e5-b466-d870c98bad0d admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:52.462929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb6f1c6-0510-49e5-b466-d870c98bad0d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:36:52.465503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb6f1c6-0510-49e5-b466-d870c98bad0d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:52.466444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb6f1c6-0510-49e5-b466-d870c98bad0d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:52.486551 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 931/1858] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:36:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:36:52.487790 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec8090b-c228-4bfd-831f-442e1a0a07df None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:52.499741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec8090b-c228-4bfd-831f-442e1a0a07df None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:52.513026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc9a0ac-f7fb-4033-b494-bcc5066353f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:52.513795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc9a0ac-f7fb-4033-b494-bcc5066353f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:52.514088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc9a0ac-f7fb-4033-b494-bcc5066353f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:52.514449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc9a0ac-f7fb-4033-b494-bcc5066353f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:52.514999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc9a0ac-f7fb-4033-b494-bcc5066353f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:52.517105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc9a0ac-f7fb-4033-b494-bcc5066353f0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:52.578713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 928/1859] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:52 2022] POST /identity/v3/projects => generated 385 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:52.582979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb6f1c6-0510-49e5-b466-d870c98bad0d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:52.597076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb6f1c6-0510-49e5-b466-d870c98bad0d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:52.726987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73718f29-a162-40dc-8100-e4065508ecbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:52.728466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73718f29-a162-40dc-8100-e4065508ecbb admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:52.728994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73718f29-a162-40dc-8100-e4065508ecbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:52.729457 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73718f29-a162-40dc-8100-e4065508ecbb admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:52.730451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73718f29-a162-40dc-8100-e4065508ecbb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:52.733550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73718f29-a162-40dc-8100-e4065508ecbb admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:52.799300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 932/1860] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:52 2022] POST /identity/v3/users => generated 293 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:52.800651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc9a0ac-f7fb-4033-b494-bcc5066353f0 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:52.816523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc9a0ac-f7fb-4033-b494-bcc5066353f0 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:52.825439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1970a4d8-aeaa-43e0-867b-14f5f9f08a41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:52.826226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1970a4d8-aeaa-43e0-867b-14f5f9f08a41 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:52.826507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1970a4d8-aeaa-43e0-867b-14f5f9f08a41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:52.826799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1970a4d8-aeaa-43e0-867b-14f5f9f08a41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:52.827451 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1970a4d8-aeaa-43e0-867b-14f5f9f08a41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:52.829194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1970a4d8-aeaa-43e0-867b-14f5f9f08a41 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:52.845296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73718f29-a162-40dc-8100-e4065508ecbb admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:52.850864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 929/1861] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:52.857231 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73718f29-a162-40dc-8100-e4065508ecbb admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:52.867377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5c4153-bc76-4fbf-94e1-eab21540294e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:52.868192 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c4153-bc76-4fbf-94e1-eab21540294e admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:52.868511 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c4153-bc76-4fbf-94e1-eab21540294e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:52.868786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5c4153-bc76-4fbf-94e1-eab21540294e admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:52.869452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5c4153-bc76-4fbf-94e1-eab21540294e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:52.871501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea5c4153-bc76-4fbf-94e1-eab21540294e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:52.878486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:52.892494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1970a4d8-aeaa-43e0-867b-14f5f9f08a41 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:52.908625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 933/1862] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:52.908743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1970a4d8-aeaa-43e0-867b-14f5f9f08a41 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:52.924582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17c4aec-a131-4d39-8d2f-5ad772ac44ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:52.925266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c4aec-a131-4d39-8d2f-5ad772ac44ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:52.925551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c4aec-a131-4d39-8d2f-5ad772ac44ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:52.925827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c4aec-a131-4d39-8d2f-5ad772ac44ae admin admin] PATH_INFO: `/v3/projects/0aa60074f54b401e80b40aeeb4c42ead/users/b64ae604593648f5bc232d3117441060/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:52.943900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17c4aec-a131-4d39-8d2f-5ad772ac44ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=0aa60074f54b401e80b40aeeb4c42ead, user_id=b64ae604593648f5bc232d3117441060, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:52.946950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17c4aec-a131-4d39-8d2f-5ad772ac44ae admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:52.965149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 930/1863] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:52 2022] PUT /identity/v3/projects/0aa60074f54b401e80b40aeeb4c42ead/users/b64ae604593648f5bc232d3117441060/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:52.978563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b7d6aa-2250-4972-a2ef-612677055474 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:52.978741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b7d6aa-2250-4972-a2ef-612677055474 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:52.978989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b7d6aa-2250-4972-a2ef-612677055474 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.013812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-00b7d6aa-2250-4972-a2ef-612677055474 None None] MFA Rules not processed for user `b64ae604593648f5bc232d3117441060`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:53.015856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-00b7d6aa-2250-4972-a2ef-612677055474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:53.091390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 934/1864] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:53.095892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17c4aec-a131-4d39-8d2f-5ad772ac44ae admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.163105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17c4aec-a131-4d39-8d2f-5ad772ac44ae admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.168226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b36eeda-3337-4318-a7f5-288f38287d2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.168873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b36eeda-3337-4318-a7f5-288f38287d2f admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.169160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b36eeda-3337-4318-a7f5-288f38287d2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.169440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b36eeda-3337-4318-a7f5-288f38287d2f admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.170116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b36eeda-3337-4318-a7f5-288f38287d2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.172207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b36eeda-3337-4318-a7f5-288f38287d2f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.201011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 931/1865] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:53 2022] POST /identity/v3/projects => generated 387 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:53.205033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b7d6aa-2250-4972-a2ef-612677055474 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.216725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b7d6aa-2250-4972-a2ef-612677055474 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.274698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755ca3eb-a64d-494b-8b9b-1157322a8ffa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.275368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755ca3eb-a64d-494b-8b9b-1157322a8ffa admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.275450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755ca3eb-a64d-494b-8b9b-1157322a8ffa admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.275854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755ca3eb-a64d-494b-8b9b-1157322a8ffa admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.276550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755ca3eb-a64d-494b-8b9b-1157322a8ffa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.278484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755ca3eb-a64d-494b-8b9b-1157322a8ffa admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.302776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 935/1866] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:53 2022] POST /identity/v3/users => generated 294 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:53.303997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b36eeda-3337-4318-a7f5-288f38287d2f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.314716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b36eeda-3337-4318-a7f5-288f38287d2f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.321041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9f6d58-8d1f-43c0-965c-20663d1cb3f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.321377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9f6d58-8d1f-43c0-965c-20663d1cb3f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.321651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9f6d58-8d1f-43c0-965c-20663d1cb3f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.322008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9f6d58-8d1f-43c0-965c-20663d1cb3f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.322558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9f6d58-8d1f-43c0-965c-20663d1cb3f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.324786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9f6d58-8d1f-43c0-965c-20663d1cb3f0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.337896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 932/1867] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:53.342598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755ca3eb-a64d-494b-8b9b-1157322a8ffa admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.358655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755ca3eb-a64d-494b-8b9b-1157322a8ffa admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.383717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69261a5-0dbe-4654-b46d-5d7ef6b77407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.384375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69261a5-0dbe-4654-b46d-5d7ef6b77407 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.384663 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69261a5-0dbe-4654-b46d-5d7ef6b77407 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.385445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69261a5-0dbe-4654-b46d-5d7ef6b77407 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.386166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69261a5-0dbe-4654-b46d-5d7ef6b77407 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.387967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69261a5-0dbe-4654-b46d-5d7ef6b77407 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.400151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 936/1868] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:36:53.404096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9f6d58-8d1f-43c0-965c-20663d1cb3f0 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.415294 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9f6d58-8d1f-43c0-965c-20663d1cb3f0 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.420191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56913029-6b6d-4a1c-9e8f-9cf0f718fd43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.420990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56913029-6b6d-4a1c-9e8f-9cf0f718fd43 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.421274 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56913029-6b6d-4a1c-9e8f-9cf0f718fd43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.421550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56913029-6b6d-4a1c-9e8f-9cf0f718fd43 admin admin] PATH_INFO: `/v3/projects/2dee5286848e4dff8ec448685c2ecf3d/users/041232186cb14ae082e85f4e971f5ebb/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.436674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56913029-6b6d-4a1c-9e8f-9cf0f718fd43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dee5286848e4dff8ec448685c2ecf3d, user_id=041232186cb14ae082e85f4e971f5ebb, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.439363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56913029-6b6d-4a1c-9e8f-9cf0f718fd43 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.453490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 933/1869] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:53 2022] PUT /identity/v3/projects/2dee5286848e4dff8ec448685c2ecf3d/users/041232186cb14ae082e85f4e971f5ebb/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:53.465157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb4b84-ef46-4593-8896-d3f313a0f8df None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.465442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb4b84-ef46-4593-8896-d3f313a0f8df None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.466286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bb4b84-ef46-4593-8896-d3f313a0f8df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.494951 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-36bb4b84-ef46-4593-8896-d3f313a0f8df None None] MFA Rules not processed for user `041232186cb14ae082e85f4e971f5ebb`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:53.496860 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-36bb4b84-ef46-4593-8896-d3f313a0f8df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:53.558835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:53.564987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 937/1870] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:53 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:53.569446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56913029-6b6d-4a1c-9e8f-9cf0f718fd43 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.575266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:53.632645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56913029-6b6d-4a1c-9e8f-9cf0f718fd43 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.637362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e3fcdd-7d98-4b10-854b-afd1bf16d3f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.637957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e3fcdd-7d98-4b10-854b-afd1bf16d3f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.638238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e3fcdd-7d98-4b10-854b-afd1bf16d3f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.638509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e3fcdd-7d98-4b10-854b-afd1bf16d3f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.639146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e3fcdd-7d98-4b10-854b-afd1bf16d3f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.641106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e3fcdd-7d98-4b10-854b-afd1bf16d3f3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.668890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 934/1871] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:36:53 2022] POST /identity/v3/projects => generated 387 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:53.673019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb4b84-ef46-4593-8896-d3f313a0f8df None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.684103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bb4b84-ef46-4593-8896-d3f313a0f8df None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.742444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ef1b3d-b2ce-48de-a8a5-5e94084131a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.743061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ef1b3d-b2ce-48de-a8a5-5e94084131a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.743347 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ef1b3d-b2ce-48de-a8a5-5e94084131a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.743615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ef1b3d-b2ce-48de-a8a5-5e94084131a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.744260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ef1b3d-b2ce-48de-a8a5-5e94084131a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.746429 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ef1b3d-b2ce-48de-a8a5-5e94084131a6 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.770205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 938/1872] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:36:53 2022] POST /identity/v3/users => generated 300 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:36:53.772265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e3fcdd-7d98-4b10-854b-afd1bf16d3f3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.783301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e3fcdd-7d98-4b10-854b-afd1bf16d3f3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.788018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c6c930-16ee-4a7a-a397-7055b8a830c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.789282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6c930-16ee-4a7a-a397-7055b8a830c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.789597 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6c930-16ee-4a7a-a397-7055b8a830c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.789819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6c930-16ee-4a7a-a397-7055b8a830c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.790452 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c6c930-16ee-4a7a-a397-7055b8a830c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.792038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c6c930-16ee-4a7a-a397-7055b8a830c4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.802825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 935/1873] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:53.807053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ef1b3d-b2ce-48de-a8a5-5e94084131a6 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.817538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ef1b3d-b2ce-48de-a8a5-5e94084131a6 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.822094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53346f89-fc03-4e29-a3e9-6c75ddc80324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.822746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53346f89-fc03-4e29-a3e9-6c75ddc80324 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.823031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53346f89-fc03-4e29-a3e9-6c75ddc80324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.823559 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53346f89-fc03-4e29-a3e9-6c75ddc80324 admin admin] PATH_INFO: `/v3/projects/e211e437990541949d4994559f1f323a/users/4ab59ff0a7bf41febccc7a7cb7be2651/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.839840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53346f89-fc03-4e29-a3e9-6c75ddc80324 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e211e437990541949d4994559f1f323a, user_id=4ab59ff0a7bf41febccc7a7cb7be2651, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.842580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53346f89-fc03-4e29-a3e9-6c75ddc80324 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.855962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 939/1874] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:53 2022] PUT /identity/v3/projects/e211e437990541949d4994559f1f323a/users/4ab59ff0a7bf41febccc7a7cb7be2651/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:53.859997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c6c930-16ee-4a7a-a397-7055b8a830c4 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.870410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c6c930-16ee-4a7a-a397-7055b8a830c4 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.936502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34caf78-c463-4123-9669-f9e09df7b5ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.937120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34caf78-c463-4123-9669-f9e09df7b5ab admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.937399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34caf78-c463-4123-9669-f9e09df7b5ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.937671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34caf78-c463-4123-9669-f9e09df7b5ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.938323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34caf78-c463-4123-9669-f9e09df7b5ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.939912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34caf78-c463-4123-9669-f9e09df7b5ab admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.952379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 936/1875] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:36:53.954863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53346f89-fc03-4e29-a3e9-6c75ddc80324 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:53.965148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53346f89-fc03-4e29-a3e9-6c75ddc80324 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:53.969734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17fa842-8155-423c-b854-e43af988707c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:53.970376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17fa842-8155-423c-b854-e43af988707c admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.970659 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17fa842-8155-423c-b854-e43af988707c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.970938 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17fa842-8155-423c-b854-e43af988707c admin admin] PATH_INFO: `/v3/projects/e211e437990541949d4994559f1f323a/users/4ab59ff0a7bf41febccc7a7cb7be2651/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:53.972886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17fa842-8155-423c-b854-e43af988707c admin admin] RBAC: Authorizing `identity:create_grant(project_id=e211e437990541949d4994559f1f323a, user_id=4ab59ff0a7bf41febccc7a7cb7be2651, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:53.975515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17fa842-8155-423c-b854-e43af988707c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:53.986206 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 940/1876] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:36:53 2022] PUT /identity/v3/projects/e211e437990541949d4994559f1f323a/users/4ab59ff0a7bf41febccc7a7cb7be2651/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:53.996973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2094c979-ff84-44ed-80e1-8a8ad4508e96 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:53.997260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2094c979-ff84-44ed-80e1-8a8ad4508e96 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:53.997538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2094c979-ff84-44ed-80e1-8a8ad4508e96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:54.024792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2094c979-ff84-44ed-80e1-8a8ad4508e96 None None] MFA Rules not processed for user `4ab59ff0a7bf41febccc7a7cb7be2651`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:54.026114 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2094c979-ff84-44ed-80e1-8a8ad4508e96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:54.100982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:54.107076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 937/1877] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:53 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:54.115487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17fa842-8155-423c-b854-e43af988707c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:54.117271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f17fa842-8155-423c-b854-e43af988707c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:54.175051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17fa842-8155-423c-b854-e43af988707c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:54.178619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da57dd9-cce0-4f7f-8a26-e5c4ae6b0a4b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:54.179479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da57dd9-cce0-4f7f-8a26-e5c4ae6b0a4b service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:54.179762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da57dd9-cce0-4f7f-8a26-e5c4ae6b0a4b service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:54.180036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da57dd9-cce0-4f7f-8a26-e5c4ae6b0a4b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:54.190424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da57dd9-cce0-4f7f-8a26-e5c4ae6b0a4b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:54.191013 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da57dd9-cce0-4f7f-8a26-e5c4ae6b0a4b service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:54.201885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 941/1878] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:54 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:54.753235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2094c979-ff84-44ed-80e1-8a8ad4508e96 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:54.764402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2094c979-ff84-44ed-80e1-8a8ad4508e96 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:54.768069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4930a039-2055-40f0-a208-1844d4bb9113 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:54.768716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4930a039-2055-40f0-a208-1844d4bb9113 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:54.769141 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4930a039-2055-40f0-a208-1844d4bb9113 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:54.769418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4930a039-2055-40f0-a208-1844d4bb9113 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:54.827216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4930a039-2055-40f0-a208-1844d4bb9113 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:54.828694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4930a039-2055-40f0-a208-1844d4bb9113 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:54.838600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 938/1879] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:54 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:55.629763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da57dd9-cce0-4f7f-8a26-e5c4ae6b0a4b service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:55.631291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4da57dd9-cce0-4f7f-8a26-e5c4ae6b0a4b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:55.704421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:55.735541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da57dd9-cce0-4f7f-8a26-e5c4ae6b0a4b service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:55.742131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:55.743057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:55.743338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:55.743610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] PATH_INFO: `/v3/users/779eabdd826e499381d890564378b93b` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:55.744982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] RBAC: Authorizing `identity:delete_user(user_id=779eabdd826e499381d890564378b93b)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:55.747246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:55.809554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '779eabdd826e499381d890564378b93b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:55.822166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '779eabdd826e499381d890564378b93b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:55.823079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '779eabdd826e499381d890564378b93b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:55.824183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '779eabdd826e499381d890564378b93b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:55.841105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '779eabdd826e499381d890564378b93b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:55.871282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 942/1880] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:55 2022] DELETE /identity/v3/users/779eabdd826e499381d890564378b93b => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:55.876293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f56905-90bb-4dfa-b160-8ed4e73d1729 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:55.876779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f56905-90bb-4dfa-b160-8ed4e73d1729 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:55.877038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f56905-90bb-4dfa-b160-8ed4e73d1729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:55.913749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f7f56905-90bb-4dfa-b160-8ed4e73d1729 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:55.915379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f7f56905-90bb-4dfa-b160-8ed4e73d1729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:55.997900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 939/1881] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:36:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:36:56.157084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:56.159189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:56.339968 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c132f5a8-7d8b-4bd4-884a-b154f8fa218a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:56.346223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02a1fe3-67ee-49c2-992b-74bdd389b75f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:56.347257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02a1fe3-67ee-49c2-992b-74bdd389b75f service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:56.347998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02a1fe3-67ee-49c2-992b-74bdd389b75f service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:56.348531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02a1fe3-67ee-49c2-992b-74bdd389b75f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:56.366564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02a1fe3-67ee-49c2-992b-74bdd389b75f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:56.370425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02a1fe3-67ee-49c2-992b-74bdd389b75f service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:56.386820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 943/1882] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:57.249808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec457446-6063-4bd1-b4a4-e0e5e714d4e7 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:57.250360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec457446-6063-4bd1-b4a4-e0e5e714d4e7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:57.250360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec457446-6063-4bd1-b4a4-e0e5e714d4e7 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:57.251711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 940/1883] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:36:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:57.337812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ded0b2-5b11-4963-9c23-c786418bafd6 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:57.340366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ded0b2-5b11-4963-9c23-c786418bafd6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:57.341119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ded0b2-5b11-4963-9c23-c786418bafd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:57.383754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-11ded0b2-5b11-4963-9c23-c786418bafd6 None None] MFA Rules not processed for user `427016f8da514dc4bbed21de08207dbb`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:57.385292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-11ded0b2-5b11-4963-9c23-c786418bafd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:57.485608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 944/1884] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:36:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:57.513934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5e4a4e-18a0-4d86-ab32-5f61b6d3fff9 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:57.514581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5e4a4e-18a0-4d86-ab32-5f61b6d3fff9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:57.514866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5e4a4e-18a0-4d86-ab32-5f61b6d3fff9 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:57.516379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 941/1885] 10.222.0.25 () {54 vars in 961 bytes} [Mon Sep 26 01:36:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:57.525316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabb956-c630-48af-9027-3f73dc836d39 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:57.526590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabb956-c630-48af-9027-3f73dc836d39 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:57.527123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabb956-c630-48af-9027-3f73dc836d39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:57.558802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-4fabb956-c630-48af-9027-3f73dc836d39 None None] MFA Rules not processed for user `427016f8da514dc4bbed21de08207dbb`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:57.560558 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4fabb956-c630-48af-9027-3f73dc836d39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:57.597987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 945/1886] 10.222.0.25 () {58 vars in 1058 bytes} [Mon Sep 26 01:36:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:57.621433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5e4a4e-18a0-4d86-ab32-5f61b6d3fff9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:57.670752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5e4a4e-18a0-4d86-ab32-5f61b6d3fff9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:57.679457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05dd4a7-cf88-466a-aeb6-ae33d320af40 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:57.680385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05dd4a7-cf88-466a-aeb6-ae33d320af40 service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:57.680883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05dd4a7-cf88-466a-aeb6-ae33d320af40 service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:57.681613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05dd4a7-cf88-466a-aeb6-ae33d320af40 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:57.693615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05dd4a7-cf88-466a-aeb6-ae33d320af40 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:57.696163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05dd4a7-cf88-466a-aeb6-ae33d320af40 service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:57.712943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 942/1887] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:58.364566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fabb956-c630-48af-9027-3f73dc836d39 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:58.366050 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4fabb956-c630-48af-9027-3f73dc836d39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:58.375830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:58.382421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fabb956-c630-48af-9027-3f73dc836d39 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:58.390948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:58.392514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:58.393068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:58.393300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] PATH_INFO: `/v3/users/5184f8f717d447cfbb7bab80d4cf000b` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:58.394779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5184f8f717d447cfbb7bab80d4cf000b)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:58.397380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:58.459081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5184f8f717d447cfbb7bab80d4cf000b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:58.467131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5184f8f717d447cfbb7bab80d4cf000b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:58.467834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5184f8f717d447cfbb7bab80d4cf000b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:58.468509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5184f8f717d447cfbb7bab80d4cf000b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:58.484601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5184f8f717d447cfbb7bab80d4cf000b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:58.500385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 946/1888] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:58 2022] DELETE /identity/v3/users/5184f8f717d447cfbb7bab80d4cf000b => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:58.685071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05dd4a7-cf88-466a-aeb6-ae33d320af40 service placement] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:58.686736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c05dd4a7-cf88-466a-aeb6-ae33d320af40 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:58.713038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05dd4a7-cf88-466a-aeb6-ae33d320af40 service placement] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:58.725193 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:58.728692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:58.842896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:58.843530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:58.844189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:58.844807 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] PATH_INFO: `/v3/projects/9a2313534557434d8e90ae45f2ed7d9b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:58.846130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a2313534557434d8e90ae45f2ed7d9b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:58.848234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:58.875669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900068d3-db69-429c-83ef-a15eb16a2f9d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:58.884451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:58.885154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:58.885438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:58.885859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] PATH_INFO: `/v3/projects/513260c0f2484a2d885d0e0a66f4e174` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:58.887078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] RBAC: Authorizing `identity:delete_project(project_id=513260c0f2484a2d885d0e0a66f4e174)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:58.889372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:58.889825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:58.909685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a2313534557434d8e90ae45f2ed7d9b', 'request_id': 'req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:58.917992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a2313534557434d8e90ae45f2ed7d9b', 'request_id': 'req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:58.931397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a2313534557434d8e90ae45f2ed7d9b', 'request_id': 'req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:58.948033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] The token cache is being invalidate because project 9a2313534557434d8e90ae45f2ed7d9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:58.949058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:58.950231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:58.954148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 943/1889] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:58 2022] DELETE /identity/v3/projects/9a2313534557434d8e90ae45f2ed7d9b => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:58.958942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:58.965389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:59.008376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '513260c0f2484a2d885d0e0a66f4e174', 'request_id': 'req-be16d6a7-fe04-41d4-942d-57fdd15a7414'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.018243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '513260c0f2484a2d885d0e0a66f4e174', 'request_id': 'req-be16d6a7-fe04-41d4-942d-57fdd15a7414'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.028092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '513260c0f2484a2d885d0e0a66f4e174', 'request_id': 'req-be16d6a7-fe04-41d4-942d-57fdd15a7414'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.056093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] The token cache is being invalidate because project 513260c0f2484a2d885d0e0a66f4e174 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:36:59.056935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.059903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.067186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 947/1890] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:58 2022] DELETE /identity/v3/projects/513260c0f2484a2d885d0e0a66f4e174 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:59.068160 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:59.070713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:59.088697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:59.130134 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3131f6-a2ed-4e0f-9d41-0ede93f1ef78 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:59.230352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be16d6a7-fe04-41d4-942d-57fdd15a7414 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:59.236920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:59.237742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:59.238131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:59.238644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] PATH_INFO: `/v3/users/3c9ec468ab4c477dab5a08cd8c6d80ac` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:59.240280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c9ec468ab4c477dab5a08cd8c6d80ac)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:59.242706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:59.245399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:59.246931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:59.247226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:59.247792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] PATH_INFO: `/v3/users/c23880d8453c4acb9c8b9278db573799` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:59.251946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c23880d8453c4acb9c8b9278db573799)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:59.259287 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:59.340760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c9ec468ab4c477dab5a08cd8c6d80ac'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.359561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c9ec468ab4c477dab5a08cd8c6d80ac'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.363190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c9ec468ab4c477dab5a08cd8c6d80ac'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.365420 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c9ec468ab4c477dab5a08cd8c6d80ac'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.410168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c23880d8453c4acb9c8b9278db573799'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.421120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a58e7c7d-3de5-44e3-ab61-806a1aafcc4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c9ec468ab4c477dab5a08cd8c6d80ac'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.429082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c23880d8453c4acb9c8b9278db573799'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.432903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c23880d8453c4acb9c8b9278db573799'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.436770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c23880d8453c4acb9c8b9278db573799'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.450830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fe202fd2-6035-4264-a771-6c087bcd33b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c23880d8453c4acb9c8b9278db573799'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:36:59.453791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:59.465022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 948/1891] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:59 2022] DELETE /identity/v3/users/3c9ec468ab4c477dab5a08cd8c6d80ac => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:59.520963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 944/1892] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:36:58 2022] DELETE /identity/v3/users/c23880d8453c4acb9c8b9278db573799 => generated 0 bytes in 560 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:36:59.543462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755036f0-ceb3-4a1a-a55f-4f16c99b8528 None None] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:59.543936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755036f0-ceb3-4a1a-a55f-4f16c99b8528 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:59.544648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755036f0-ceb3-4a1a-a55f-4f16c99b8528 None None] PATH_INFO: `/` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:59.557373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 949/1893] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:36:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:36:59.569335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2102025e-8b4f-4122-9b74-354a38dccfa1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:59.569335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2102025e-8b4f-4122-9b74-354a38dccfa1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:59.569335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2102025e-8b4f-4122-9b74-354a38dccfa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:59.620312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2102025e-8b4f-4122-9b74-354a38dccfa1 None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:36:59.622310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2102025e-8b4f-4122-9b74-354a38dccfa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:59.718551 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755036f0-ceb3-4a1a-a55f-4f16c99b8528 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:59.720478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-755036f0-ceb3-4a1a-a55f-4f16c99b8528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:59.805636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 945/1894] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:36:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:36:59.806671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2102025e-8b4f-4122-9b74-354a38dccfa1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:59.808615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2102025e-8b4f-4122-9b74-354a38dccfa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:59.859930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755036f0-ceb3-4a1a-a55f-4f16c99b8528 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:59.864821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00028e9-8410-4e13-aa38-19a80e37d030 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:59.865573 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00028e9-8410-4e13-aa38-19a80e37d030 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:59.865993 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00028e9-8410-4e13-aa38-19a80e37d030 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:59.866477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00028e9-8410-4e13-aa38-19a80e37d030 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:59.868717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a00028e9-8410-4e13-aa38-19a80e37d030 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:59.906307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2102025e-8b4f-4122-9b74-354a38dccfa1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:36:59.915391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:36:59.916043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:36:59.916322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:36:59.916622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] PATH_INFO: `/v3/projects/b55aac0e665b498ab290c912d8bb6526` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:36:59.918009 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=b55aac0e665b498ab290c912d8bb6526)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:59.920959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:59.927230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:36:59.957191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00028e9-8410-4e13-aa38-19a80e37d030 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:36:59.959834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00028e9-8410-4e13-aa38-19a80e37d030 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:36:59.972776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 950/1895] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:59 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:36:59.973811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00028e9-8410-4e13-aa38-19a80e37d030 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:36:59.975246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a00028e9-8410-4e13-aa38-19a80e37d030 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:36:59.998399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00028e9-8410-4e13-aa38-19a80e37d030 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:00.012735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b55aac0e665b498ab290c912d8bb6526', 'request_id': 'req-b48a1901-87b1-4168-9552-b063673569fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:00.030724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b55aac0e665b498ab290c912d8bb6526', 'request_id': 'req-b48a1901-87b1-4168-9552-b063673569fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:00.053810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b55aac0e665b498ab290c912d8bb6526', 'request_id': 'req-b48a1901-87b1-4168-9552-b063673569fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:00.073102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] The token cache is being invalidate because project b55aac0e665b498ab290c912d8bb6526 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:00.073776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:00.074515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b48a1901-87b1-4168-9552-b063673569fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:00.077166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 946/1896] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:36:59 2022] DELETE /identity/v3/projects/b55aac0e665b498ab290c912d8bb6526 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:00.081701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb715e3f-cc8e-4db5-84e6-36dbab70cca2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.083249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb715e3f-cc8e-4db5-84e6-36dbab70cca2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.083537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb715e3f-cc8e-4db5-84e6-36dbab70cca2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.096680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:00.109629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d57259-d929-4b16-81da-7c92a0613ff1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:00.109879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-eb715e3f-cc8e-4db5-84e6-36dbab70cca2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:00.111670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-eb715e3f-cc8e-4db5-84e6-36dbab70cca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:00.112819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d57259-d929-4b16-81da-7c92a0613ff1 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.113307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d57259-d929-4b16-81da-7c92a0613ff1 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.113831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d57259-d929-4b16-81da-7c92a0613ff1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.116422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-28d57259-d929-4b16-81da-7c92a0613ff1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:00.131480 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d57259-d929-4b16-81da-7c92a0613ff1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:00.134750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d57259-d929-4b16-81da-7c92a0613ff1 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:00.165345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 951/1897] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:36:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:00.193083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 947/1898] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:00.201099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97516ca4-4a9f-4313-954c-fe787cc2b799 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.201449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97516ca4-4a9f-4313-954c-fe787cc2b799 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.201885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97516ca4-4a9f-4313-954c-fe787cc2b799 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.228974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-97516ca4-4a9f-4313-954c-fe787cc2b799 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:00.230496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-97516ca4-4a9f-4313-954c-fe787cc2b799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:00.257827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 952/1899] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:00.261893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb715e3f-cc8e-4db5-84e6-36dbab70cca2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:00.276100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb715e3f-cc8e-4db5-84e6-36dbab70cca2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:00.281776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bc106e-b178-42b6-a70c-6590b8c95768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:00.282962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bc106e-b178-42b6-a70c-6590b8c95768 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.283374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bc106e-b178-42b6-a70c-6590b8c95768 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.284359 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bc106e-b178-42b6-a70c-6590b8c95768 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.285300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bc106e-b178-42b6-a70c-6590b8c95768 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:00.285719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bc106e-b178-42b6-a70c-6590b8c95768 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:00.290592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bc106e-b178-42b6-a70c-6590b8c95768 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:00.306676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 948/1900] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:37:00.311418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97516ca4-4a9f-4313-954c-fe787cc2b799 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:00.325186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97516ca4-4a9f-4313-954c-fe787cc2b799 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:00.331593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a70a7e-0ff2-4f9b-816c-253f49386612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:00.334372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a70a7e-0ff2-4f9b-816c-253f49386612 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.334626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a70a7e-0ff2-4f9b-816c-253f49386612 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.335805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a70a7e-0ff2-4f9b-816c-253f49386612 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.336627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a70a7e-0ff2-4f9b-816c-253f49386612 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:00.339339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a70a7e-0ff2-4f9b-816c-253f49386612 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:00.387053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 953/1901] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:00 2022] POST /identity/v3/projects => generated 371 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:00.392103 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bc106e-b178-42b6-a70c-6590b8c95768 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:00.407604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bc106e-b178-42b6-a70c-6590b8c95768 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:00.483749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cf99bf-af9a-4b9e-95f7-31b20f6dd2ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:00.486792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cf99bf-af9a-4b9e-95f7-31b20f6dd2ff admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.487073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cf99bf-af9a-4b9e-95f7-31b20f6dd2ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.487307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cf99bf-af9a-4b9e-95f7-31b20f6dd2ff admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.489198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cf99bf-af9a-4b9e-95f7-31b20f6dd2ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:00.491296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cf99bf-af9a-4b9e-95f7-31b20f6dd2ff admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:00.521520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 949/1902] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:00 2022] POST /identity/v3/users => generated 286 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:00.526619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a70a7e-0ff2-4f9b-816c-253f49386612 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:00.538989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a70a7e-0ff2-4f9b-816c-253f49386612 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:00.555742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bf9c18-4d1f-44d4-abab-560a88fbaba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:00.556412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bf9c18-4d1f-44d4-abab-560a88fbaba1 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.556702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bf9c18-4d1f-44d4-abab-560a88fbaba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.556983 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bf9c18-4d1f-44d4-abab-560a88fbaba1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.559798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bf9c18-4d1f-44d4-abab-560a88fbaba1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:00.562695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bf9c18-4d1f-44d4-abab-560a88fbaba1 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:00.596514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 954/1903] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:00 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:00.597934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cf99bf-af9a-4b9e-95f7-31b20f6dd2ff admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:00.610094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cf99bf-af9a-4b9e-95f7-31b20f6dd2ff admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:00.616328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d918d6c-ea86-4e07-ad52-fa9c5e91c026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:00.617576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d918d6c-ea86-4e07-ad52-fa9c5e91c026 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.618234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d918d6c-ea86-4e07-ad52-fa9c5e91c026 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.618704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d918d6c-ea86-4e07-ad52-fa9c5e91c026 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.620843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d918d6c-ea86-4e07-ad52-fa9c5e91c026 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:00.623341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d918d6c-ea86-4e07-ad52-fa9c5e91c026 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:00.639327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 950/1904] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:00 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:00.644159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bf9c18-4d1f-44d4-abab-560a88fbaba1 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:00.658263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bf9c18-4d1f-44d4-abab-560a88fbaba1 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:00.663594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf149e2-dac8-48cb-a90c-792ef71de8e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:00.665713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf149e2-dac8-48cb-a90c-792ef71de8e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.666192 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf149e2-dac8-48cb-a90c-792ef71de8e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.666662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf149e2-dac8-48cb-a90c-792ef71de8e4 admin admin] PATH_INFO: `/v3/projects/39701a8b71094a9690bde90bba69c8ae/users/d7b4ce2904fa488da41ca91412e71d8a/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.689498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf149e2-dac8-48cb-a90c-792ef71de8e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39701a8b71094a9690bde90bba69c8ae, user_id=d7b4ce2904fa488da41ca91412e71d8a, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:00.692521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf149e2-dac8-48cb-a90c-792ef71de8e4 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:00.721489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 955/1905] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:00 2022] PUT /identity/v3/projects/39701a8b71094a9690bde90bba69c8ae/users/d7b4ce2904fa488da41ca91412e71d8a/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:00.731860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f16ac6c-4dfb-45a3-983c-751fdaf3ea50 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.732319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f16ac6c-4dfb-45a3-983c-751fdaf3ea50 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.732634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f16ac6c-4dfb-45a3-983c-751fdaf3ea50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.778758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2f16ac6c-4dfb-45a3-983c-751fdaf3ea50 None None] MFA Rules not processed for user `d7b4ce2904fa488da41ca91412e71d8a`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:00.779994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2f16ac6c-4dfb-45a3-983c-751fdaf3ea50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:00.856963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:00.863246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 951/1906] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:00 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:00.888791 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf149e2-dac8-48cb-a90c-792ef71de8e4 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:00.892081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ddf149e2-dac8-48cb-a90c-792ef71de8e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:00.963293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf149e2-dac8-48cb-a90c-792ef71de8e4 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:00.973773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fac3c92-2239-49d3-b34e-4c97686c49ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:00.974620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fac3c92-2239-49d3-b34e-4c97686c49ed service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:00.975066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fac3c92-2239-49d3-b34e-4c97686c49ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:00.975521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fac3c92-2239-49d3-b34e-4c97686c49ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:00.986905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fac3c92-2239-49d3-b34e-4c97686c49ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:00.988852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fac3c92-2239-49d3-b34e-4c97686c49ed service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:01.002116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 956/1907] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:00 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:01.027521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f16ac6c-4dfb-45a3-983c-751fdaf3ea50 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:01.204879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f16ac6c-4dfb-45a3-983c-751fdaf3ea50 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:01.210656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:01.211318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:01.222132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:01.224456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] PATH_INFO: `/v3/users/d7b4ce2904fa488da41ca91412e71d8a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:01.227760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7b4ce2904fa488da41ca91412e71d8a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:01.233020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:01.391774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fac3c92-2239-49d3-b34e-4c97686c49ed service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:01.392578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1fac3c92-2239-49d3-b34e-4c97686c49ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:01.407446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7b4ce2904fa488da41ca91412e71d8a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:01.420752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7b4ce2904fa488da41ca91412e71d8a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:01.422299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7b4ce2904fa488da41ca91412e71d8a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:01.423872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:01.425533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7b4ce2904fa488da41ca91412e71d8a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:01.439537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0139826a-9a50-48dc-8372-c22ac5136f64 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7b4ce2904fa488da41ca91412e71d8a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:01.454555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fac3c92-2239-49d3-b34e-4c97686c49ed service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:01.493899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 952/1908] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:01 2022] DELETE /identity/v3/users/d7b4ce2904fa488da41ca91412e71d8a => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:01.494567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:01.495566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:01.496085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:01.496543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] PATH_INFO: `/v3/projects/e0c748b360d94e3ca417ed4eb257eb80` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:01.498623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0c748b360d94e3ca417ed4eb257eb80)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:01.502855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b668a642-d135-4411-b6c4-8a98760042bf None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:01.503520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b668a642-d135-4411-b6c4-8a98760042bf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:01.504909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b668a642-d135-4411-b6c4-8a98760042bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:01.524511 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:01.561828 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b668a642-d135-4411-b6c4-8a98760042bf None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:01.566242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b668a642-d135-4411-b6c4-8a98760042bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:01.630959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0c748b360d94e3ca417ed4eb257eb80', 'request_id': 'req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:01.643670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0c748b360d94e3ca417ed4eb257eb80', 'request_id': 'req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:01.652695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0c748b360d94e3ca417ed4eb257eb80', 'request_id': 'req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:01.677093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] The token cache is being invalidate because project e0c748b360d94e3ca417ed4eb257eb80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:01.677691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:01.678722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-bfab8aa4-de1c-4c0d-8db0-eb6b6b335f21 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:01.681664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 957/1909] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:01 2022] DELETE /identity/v3/projects/e0c748b360d94e3ca417ed4eb257eb80 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:01.686653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f74dd80-d513-4da7-b052-ad4de48970ca None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:01.686933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f74dd80-d513-4da7-b052-ad4de48970ca None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:01.688384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f74dd80-d513-4da7-b052-ad4de48970ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:01.701863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 953/1910] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:01.713862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b668a642-d135-4411-b6c4-8a98760042bf None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:01.715159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7f74dd80-d513-4da7-b052-ad4de48970ca None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:01.716595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7f74dd80-d513-4da7-b052-ad4de48970ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:01.717351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b668a642-d135-4411-b6c4-8a98760042bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:01.746884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:01.796726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b668a642-d135-4411-b6c4-8a98760042bf None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:01.805947 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 958/1911] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:01.814081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72465b8f-9510-4cf0-a927-416b06ee1e15 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:01.814308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72465b8f-9510-4cf0-a927-416b06ee1e15 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:01.814717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72465b8f-9510-4cf0-a927-416b06ee1e15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:01.844319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-72465b8f-9510-4cf0-a927-416b06ee1e15 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:01.845570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-72465b8f-9510-4cf0-a927-416b06ee1e15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:01.855994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 959/1912] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:01.860287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72465b8f-9510-4cf0-a927-416b06ee1e15 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:01.863364 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367cd61f-6484-4eb7-b995-df03a3e6bf2e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:01.863981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367cd61f-6484-4eb7-b995-df03a3e6bf2e service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:01.864264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367cd61f-6484-4eb7-b995-df03a3e6bf2e service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:01.864568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367cd61f-6484-4eb7-b995-df03a3e6bf2e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:01.870215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72465b8f-9510-4cf0-a927-416b06ee1e15 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:01.874636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367cd61f-6484-4eb7-b995-df03a3e6bf2e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:01.874817 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e94568-b68b-4b2b-a603-ebab55f38846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:01.875474 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e94568-b68b-4b2b-a603-ebab55f38846 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:01.875816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e94568-b68b-4b2b-a603-ebab55f38846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:01.876069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367cd61f-6484-4eb7-b995-df03a3e6bf2e service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:01.876189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e94568-b68b-4b2b-a603-ebab55f38846 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:01.877077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e94568-b68b-4b2b-a603-ebab55f38846 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:01.877446 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e94568-b68b-4b2b-a603-ebab55f38846 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:01.879083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e94568-b68b-4b2b-a603-ebab55f38846 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:01.886212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 954/1913] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:37:01.891400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 960/1914] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:37:01.895662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367cd61f-6484-4eb7-b995-df03a3e6bf2e service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:01.906101 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367cd61f-6484-4eb7-b995-df03a3e6bf2e service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:01.910859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf8dd4-8ff2-465c-9205-6ecaf5f80951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:01.911511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf8dd4-8ff2-465c-9205-6ecaf5f80951 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:01.911717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf8dd4-8ff2-465c-9205-6ecaf5f80951 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:01.911988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecf8dd4-8ff2-465c-9205-6ecaf5f80951 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:01.912647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf8dd4-8ff2-465c-9205-6ecaf5f80951 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:01.915484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecf8dd4-8ff2-465c-9205-6ecaf5f80951 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:01.947548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 955/1915] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:01 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:01.949338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e94568-b68b-4b2b-a603-ebab55f38846 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:01.959745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e94568-b68b-4b2b-a603-ebab55f38846 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.020088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7431c2a9-ed60-4b5e-8369-839b5ecf7161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.020710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7431c2a9-ed60-4b5e-8369-839b5ecf7161 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.021054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7431c2a9-ed60-4b5e-8369-839b5ecf7161 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.021363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7431c2a9-ed60-4b5e-8369-839b5ecf7161 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.022127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7431c2a9-ed60-4b5e-8369-839b5ecf7161 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.025026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7431c2a9-ed60-4b5e-8369-839b5ecf7161 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.046378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:02.057342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 961/1916] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:01 2022] POST /identity/v3/users => generated 293 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:02.058374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf8dd4-8ff2-465c-9205-6ecaf5f80951 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.069884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecf8dd4-8ff2-465c-9205-6ecaf5f80951 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.075019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36078926-6d31-404b-82b6-59161bcf9484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.075625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36078926-6d31-404b-82b6-59161bcf9484 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.075905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36078926-6d31-404b-82b6-59161bcf9484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.076178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36078926-6d31-404b-82b6-59161bcf9484 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.076863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36078926-6d31-404b-82b6-59161bcf9484 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.078460 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36078926-6d31-404b-82b6-59161bcf9484 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.090126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 956/1917] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:02 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:02.094376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7431c2a9-ed60-4b5e-8369-839b5ecf7161 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.105066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7431c2a9-ed60-4b5e-8369-839b5ecf7161 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.109823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52990f46-3cb4-4946-9879-5ba3646e25b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.110526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52990f46-3cb4-4946-9879-5ba3646e25b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.110879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52990f46-3cb4-4946-9879-5ba3646e25b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.111252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52990f46-3cb4-4946-9879-5ba3646e25b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.112053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52990f46-3cb4-4946-9879-5ba3646e25b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.113798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52990f46-3cb4-4946-9879-5ba3646e25b3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.124868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 962/1918] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:02.129012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36078926-6d31-404b-82b6-59161bcf9484 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.140272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36078926-6d31-404b-82b6-59161bcf9484 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.145833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffad024-4975-4495-a353-d8eaf699966d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.146562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffad024-4975-4495-a353-d8eaf699966d admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.146840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffad024-4975-4495-a353-d8eaf699966d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.147110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffad024-4975-4495-a353-d8eaf699966d admin admin] PATH_INFO: `/v3/projects/0e5203ad82e245e88d5d638c60b02d49/users/98ab328072ee4f85ac3835bc920b01ff/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.164582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffad024-4975-4495-a353-d8eaf699966d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e5203ad82e245e88d5d638c60b02d49, user_id=98ab328072ee4f85ac3835bc920b01ff, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.167266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffad024-4975-4495-a353-d8eaf699966d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.168833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52990f46-3cb4-4946-9879-5ba3646e25b3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.170757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-52990f46-3cb4-4946-9879-5ba3646e25b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:02.185813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52990f46-3cb4-4946-9879-5ba3646e25b3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.188934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 957/1919] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:02 2022] PUT /identity/v3/projects/0e5203ad82e245e88d5d638c60b02d49/users/98ab328072ee4f85ac3835bc920b01ff/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:02.206895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7b4a0c-24eb-4682-851a-8efb1daa0812 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.206895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7b4a0c-24eb-4682-851a-8efb1daa0812 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.206895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7b4a0c-24eb-4682-851a-8efb1daa0812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.240547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bf7b4a0c-24eb-4682-851a-8efb1daa0812 None None] MFA Rules not processed for user `98ab328072ee4f85ac3835bc920b01ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:02.242931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bf7b4a0c-24eb-4682-851a-8efb1daa0812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:02.273302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.273931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.274212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.274570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] PATH_INFO: `/v3/projects/39701a8b71094a9690bde90bba69c8ae` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.275606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] RBAC: Authorizing `identity:delete_project(project_id=39701a8b71094a9690bde90bba69c8ae)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.277592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.320330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 958/1920] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:02.326536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7b4a0c-24eb-4682-851a-8efb1daa0812 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.338989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7b4a0c-24eb-4682-851a-8efb1daa0812 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.349788 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:02.377986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39701a8b71094a9690bde90bba69c8ae', 'request_id': 'req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:02.386859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39701a8b71094a9690bde90bba69c8ae', 'request_id': 'req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:02.393942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39701a8b71094a9690bde90bba69c8ae', 'request_id': 'req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:02.411993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014bf0b1-1117-481e-b6e8-df4a28902871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.412697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014bf0b1-1117-481e-b6e8-df4a28902871 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.413536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014bf0b1-1117-481e-b6e8-df4a28902871 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.413642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] The token cache is being invalidate because project 39701a8b71094a9690bde90bba69c8ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:02.413931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014bf0b1-1117-481e-b6e8-df4a28902871 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.414152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:02.414569 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014bf0b1-1117-481e-b6e8-df4a28902871 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.415140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b2d1a26-a520-4595-9fcb-3cfdf357c59a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:02.416544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014bf0b1-1117-481e-b6e8-df4a28902871 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.417126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 963/1921] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:02 2022] DELETE /identity/v3/projects/39701a8b71094a9690bde90bba69c8ae => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:02.422050 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05a0aa9-e4f0-4802-a744-e9edba910d7b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.422494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05a0aa9-e4f0-4802-a744-e9edba910d7b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.422813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05a0aa9-e4f0-4802-a744-e9edba910d7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.452425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e05a0aa9-e4f0-4802-a744-e9edba910d7b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:02.453972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e05a0aa9-e4f0-4802-a744-e9edba910d7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:02.462251 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 959/1922] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:02 2022] POST /identity/v3/projects => generated 385 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:02.463274 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014bf0b1-1117-481e-b6e8-df4a28902871 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.464286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 964/1923] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:02.469541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-014bf0b1-1117-481e-b6e8-df4a28902871 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:02.473107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95875bc-8e36-4636-93a1-0c70614b9f65 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.473390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95875bc-8e36-4636-93a1-0c70614b9f65 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.473687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95875bc-8e36-4636-93a1-0c70614b9f65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.502323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b95875bc-8e36-4636-93a1-0c70614b9f65 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:02.504565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b95875bc-8e36-4636-93a1-0c70614b9f65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:02.572140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014bf0b1-1117-481e-b6e8-df4a28902871 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.585741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bff48f-32c5-46d0-b4a9-905b34b7dba8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.588269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bff48f-32c5-46d0-b4a9-905b34b7dba8 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.588729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bff48f-32c5-46d0-b4a9-905b34b7dba8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.589142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bff48f-32c5-46d0-b4a9-905b34b7dba8 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.589952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bff48f-32c5-46d0-b4a9-905b34b7dba8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.595785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bff48f-32c5-46d0-b4a9-905b34b7dba8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.597201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 965/1924] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:02.601402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95875bc-8e36-4636-93a1-0c70614b9f65 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.611118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95875bc-8e36-4636-93a1-0c70614b9f65 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.618220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31821de9-d26d-423d-96db-2b69408d3012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.619226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31821de9-d26d-423d-96db-2b69408d3012 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.619600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31821de9-d26d-423d-96db-2b69408d3012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.620402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31821de9-d26d-423d-96db-2b69408d3012 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.621385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31821de9-d26d-423d-96db-2b69408d3012 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:02.621385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31821de9-d26d-423d-96db-2b69408d3012 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.622942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31821de9-d26d-423d-96db-2b69408d3012 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.632713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:02.642586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 960/1925] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:02 2022] POST /identity/v3/users => generated 293 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:02.651864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 966/1926] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:02.652051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bff48f-32c5-46d0-b4a9-905b34b7dba8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.658804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31821de9-d26d-423d-96db-2b69408d3012 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.674045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bff48f-32c5-46d0-b4a9-905b34b7dba8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.674360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31821de9-d26d-423d-96db-2b69408d3012 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.684757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c67f05-befa-4ac7-b24b-9857e7f20429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.685929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c67f05-befa-4ac7-b24b-9857e7f20429 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.685929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c67f05-befa-4ac7-b24b-9857e7f20429 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.685929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c67f05-befa-4ac7-b24b-9857e7f20429 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.686708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c67f05-befa-4ac7-b24b-9857e7f20429 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.688675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c67f05-befa-4ac7-b24b-9857e7f20429 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.689534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7c56aa-28b9-42f6-b256-fa42a7814246 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.693827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7c56aa-28b9-42f6-b256-fa42a7814246 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.695231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7c56aa-28b9-42f6-b256-fa42a7814246 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.695708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7c56aa-28b9-42f6-b256-fa42a7814246 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.700715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7c56aa-28b9-42f6-b256-fa42a7814246 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.707814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7c56aa-28b9-42f6-b256-fa42a7814246 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.742825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 961/1927] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:02 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:02.744592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7c56aa-28b9-42f6-b256-fa42a7814246 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.760652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 967/1928] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:02 2022] POST /identity/v3/projects => generated 383 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:02.760652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c67f05-befa-4ac7-b24b-9857e7f20429 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:02.769426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7c56aa-28b9-42f6-b256-fa42a7814246 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.933475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c67f05-befa-4ac7-b24b-9857e7f20429 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:02.958273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8e26a6-2f6a-43e8-b08e-f7a6c1842647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.962986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8e26a6-2f6a-43e8-b08e-f7a6c1842647 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.963858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8e26a6-2f6a-43e8-b08e-f7a6c1842647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.964687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8e26a6-2f6a-43e8-b08e-f7a6c1842647 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.967523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8e26a6-2f6a-43e8-b08e-f7a6c1842647 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:02.977573 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8e26a6-2f6a-43e8-b08e-f7a6c1842647 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:02.986498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf175dd8-364c-44ae-a6eb-4d81d4accd55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:02.988801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf175dd8-364c-44ae-a6eb-4d81d4accd55 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:02.989093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf175dd8-364c-44ae-a6eb-4d81d4accd55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:02.989374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf175dd8-364c-44ae-a6eb-4d81d4accd55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:02.997042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf175dd8-364c-44ae-a6eb-4d81d4accd55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.000606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf175dd8-364c-44ae-a6eb-4d81d4accd55 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.037324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 962/1929] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:02 2022] GET /identity/v3/roles => generated 1420 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:03.044615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf175dd8-364c-44ae-a6eb-4d81d4accd55 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.061918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 968/1930] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:02 2022] POST /identity/v3/users => generated 292 bytes in 307 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:03.062890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8e26a6-2f6a-43e8-b08e-f7a6c1842647 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.080594 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf175dd8-364c-44ae-a6eb-4d81d4accd55 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.084458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8e26a6-2f6a-43e8-b08e-f7a6c1842647 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.091102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2d1cab-b1b8-431d-a783-aa5c2775163c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.091778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2d1cab-b1b8-431d-a783-aa5c2775163c admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.092746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2d1cab-b1b8-431d-a783-aa5c2775163c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.093035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2d1cab-b1b8-431d-a783-aa5c2775163c admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.093779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2d1cab-b1b8-431d-a783-aa5c2775163c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.095411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2d1cab-b1b8-431d-a783-aa5c2775163c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.103890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c44be10-5528-4624-a99b-6a9f9da1df18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.106527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c44be10-5528-4624-a99b-6a9f9da1df18 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.106814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c44be10-5528-4624-a99b-6a9f9da1df18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.107115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c44be10-5528-4624-a99b-6a9f9da1df18 admin admin] PATH_INFO: `/v3/projects/c5320b0d28a145bf98d4fab4d73e6edb/users/28c7737926604698ae2defb96f83434b/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.135334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 969/1931] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:03 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:03.135529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:03.137531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2d1cab-b1b8-431d-a783-aa5c2775163c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.148268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c44be10-5528-4624-a99b-6a9f9da1df18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5320b0d28a145bf98d4fab4d73e6edb, user_id=28c7737926604698ae2defb96f83434b, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.151233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c44be10-5528-4624-a99b-6a9f9da1df18 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.164987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2d1cab-b1b8-431d-a783-aa5c2775163c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.180650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dc157c-92d9-4ce3-aaad-2606e8cb20ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.181438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dc157c-92d9-4ce3-aaad-2606e8cb20ca admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.181854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dc157c-92d9-4ce3-aaad-2606e8cb20ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.182278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dc157c-92d9-4ce3-aaad-2606e8cb20ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.183440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 963/1932] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:03 2022] PUT /identity/v3/projects/c5320b0d28a145bf98d4fab4d73e6edb/users/28c7737926604698ae2defb96f83434b/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:03.183794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dc157c-92d9-4ce3-aaad-2606e8cb20ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.185924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dc157c-92d9-4ce3-aaad-2606e8cb20ca admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.199118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35819be-8849-496b-81ee-e81724d87045 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.199770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35819be-8849-496b-81ee-e81724d87045 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.200461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35819be-8849-496b-81ee-e81724d87045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.215730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 970/1933] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:03 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:03.219888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dc157c-92d9-4ce3-aaad-2606e8cb20ca admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.233611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dc157c-92d9-4ce3-aaad-2606e8cb20ca admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.248431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a35819be-8849-496b-81ee-e81724d87045 None None] MFA Rules not processed for user `28c7737926604698ae2defb96f83434b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:03.250654 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a35819be-8849-496b-81ee-e81724d87045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:03.362020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81daa46a-6c99-4be1-8597-2f9c817bbe09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.362715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81daa46a-6c99-4be1-8597-2f9c817bbe09 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.362999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81daa46a-6c99-4be1-8597-2f9c817bbe09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.363279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81daa46a-6c99-4be1-8597-2f9c817bbe09 admin admin] PATH_INFO: `/v3/projects/01a95b064cf4490bab78eb2fa2fd026d/users/ba818bad3e4948a1820ef8f825ac4078/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.387381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 964/1934] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:03.387381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35819be-8849-496b-81ee-e81724d87045 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.387681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81daa46a-6c99-4be1-8597-2f9c817bbe09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=01a95b064cf4490bab78eb2fa2fd026d, user_id=ba818bad3e4948a1820ef8f825ac4078, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.387776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a35819be-8849-496b-81ee-e81724d87045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:03.389185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81daa46a-6c99-4be1-8597-2f9c817bbe09 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.409246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 971/1935] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:03 2022] PUT /identity/v3/projects/01a95b064cf4490bab78eb2fa2fd026d/users/ba818bad3e4948a1820ef8f825ac4078/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:03.410971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81daa46a-6c99-4be1-8597-2f9c817bbe09 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.420038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35819be-8849-496b-81ee-e81724d87045 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.468225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:03.515407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81daa46a-6c99-4be1-8597-2f9c817bbe09 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.520724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.520874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadee088-16b5-4801-bc9a-11bf3081ba96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.521437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadee088-16b5-4801-bc9a-11bf3081ba96 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.521576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.521730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadee088-16b5-4801-bc9a-11bf3081ba96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.522010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadee088-16b5-4801-bc9a-11bf3081ba96 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.522111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.522646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eadee088-16b5-4801-bc9a-11bf3081ba96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.522766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] PATH_INFO: `/v3/users/b64ae604593648f5bc232d3117441060` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.524188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b64ae604593648f5bc232d3117441060)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.525298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eadee088-16b5-4801-bc9a-11bf3081ba96 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.526789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.583665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 972/1936] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:03 2022] POST /identity/v3/projects => generated 383 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:03.585332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45e0df-c27a-4ee6-921c-c3ddff6f6922 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.585983 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45e0df-c27a-4ee6-921c-c3ddff6f6922 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.586232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45e0df-c27a-4ee6-921c-c3ddff6f6922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.607716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b64ae604593648f5bc232d3117441060'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:03.617198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b64ae604593648f5bc232d3117441060'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:03.617971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b64ae604593648f5bc232d3117441060'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:03.618604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b64ae604593648f5bc232d3117441060'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:03.620599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-1e45e0df-c27a-4ee6-921c-c3ddff6f6922 None None] MFA Rules not processed for user `ba818bad3e4948a1820ef8f825ac4078`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:03.621965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1e45e0df-c27a-4ee6-921c-c3ddff6f6922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:03.626988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b64ae604593648f5bc232d3117441060'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:03.639454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:03.647203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 965/1937] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:03 2022] DELETE /identity/v3/users/b64ae604593648f5bc232d3117441060 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:03.648912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.651621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:03.689923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 973/1938] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:03 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:03.691170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f59794-c1e0-4bae-8e1e-a9902229ea85 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.691456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f59794-c1e0-4bae-8e1e-a9902229ea85 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.691744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f59794-c1e0-4bae-8e1e-a9902229ea85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.715812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e6f59794-c1e0-4bae-8e1e-a9902229ea85 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:03.717485 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e6f59794-c1e0-4bae-8e1e-a9902229ea85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:03.732864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fb7b7c-999f-40be-8a02-e123450e99d2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.739681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5e75a6-0970-42ee-ad56-01b6b40bc474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.740289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e75a6-0970-42ee-ad56-01b6b40bc474 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.740595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e75a6-0970-42ee-ad56-01b6b40bc474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.740867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e75a6-0970-42ee-ad56-01b6b40bc474 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.742230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5e75a6-0970-42ee-ad56-01b6b40bc474 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.744442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5e75a6-0970-42ee-ad56-01b6b40bc474 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.774286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 966/1939] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:03 2022] POST /identity/v3/users => generated 298 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:03.775506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5e75a6-0970-42ee-ad56-01b6b40bc474 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.777330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bd5e75a6-0970-42ee-ad56-01b6b40bc474 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:03.791035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 974/1940] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:03.791997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f59794-c1e0-4bae-8e1e-a9902229ea85 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.802926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f59794-c1e0-4bae-8e1e-a9902229ea85 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.808870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff2e9af-e880-48d9-a76e-e68860955e87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.809483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff2e9af-e880-48d9-a76e-e68860955e87 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.809761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff2e9af-e880-48d9-a76e-e68860955e87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.810032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff2e9af-e880-48d9-a76e-e68860955e87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.810680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff2e9af-e880-48d9-a76e-e68860955e87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.812290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff2e9af-e880-48d9-a76e-e68860955e87 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.823773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 975/1941] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:03.827650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff2e9af-e880-48d9-a76e-e68860955e87 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.829394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2ff2e9af-e880-48d9-a76e-e68860955e87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:03.851148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5e75a6-0970-42ee-ad56-01b6b40bc474 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.854800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ca0fba-be72-4b73-b973-45787a16e34b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.855408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ca0fba-be72-4b73-b973-45787a16e34b service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.855690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ca0fba-be72-4b73-b973-45787a16e34b service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.855953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ca0fba-be72-4b73-b973-45787a16e34b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.870049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:03.894442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff2e9af-e880-48d9-a76e-e68860955e87 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.898801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01155c5f-bab6-49ed-ba8b-7b090a89d02b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.899411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01155c5f-bab6-49ed-ba8b-7b090a89d02b service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.899688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01155c5f-bab6-49ed-ba8b-7b090a89d02b service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.899963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01155c5f-bab6-49ed-ba8b-7b090a89d02b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.909338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01155c5f-bab6-49ed-ba8b-7b090a89d02b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.910777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01155c5f-bab6-49ed-ba8b-7b090a89d02b service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.920916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 976/1942] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:03.921706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01155c5f-bab6-49ed-ba8b-7b090a89d02b service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:03.929259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ca0fba-be72-4b73-b973-45787a16e34b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.931136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ca0fba-be72-4b73-b973-45787a16e34b service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:03.932586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01155c5f-bab6-49ed-ba8b-7b090a89d02b service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:03.939452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927bacb0-2002-443c-9728-6eeeeb705750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:03.939781 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927bacb0-2002-443c-9728-6eeeeb705750 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:03.940062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927bacb0-2002-443c-9728-6eeeeb705750 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:03.940332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927bacb0-2002-443c-9728-6eeeeb705750 admin admin] PATH_INFO: `/v3/projects/c61ce3bb85cd454a80fc3ba695875284/users/b0b78ee0114341738201d8fb6b583677/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:03.942045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 967/1943] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:03 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:03.979391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927bacb0-2002-443c-9728-6eeeeb705750 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c61ce3bb85cd454a80fc3ba695875284, user_id=b0b78ee0114341738201d8fb6b583677, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:03.984553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927bacb0-2002-443c-9728-6eeeeb705750 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:04.005336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 977/1944] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:03 2022] PUT /identity/v3/projects/c61ce3bb85cd454a80fc3ba695875284/users/b0b78ee0114341738201d8fb6b583677/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:04.010047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ca0fba-be72-4b73-b973-45787a16e34b service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:04.023815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ca0fba-be72-4b73-b973-45787a16e34b service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:04.095530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:04.122818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e645ad9b-58be-4c30-9b14-e0dbe1666550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:04.123710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e645ad9b-58be-4c30-9b14-e0dbe1666550 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:04.124635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e645ad9b-58be-4c30-9b14-e0dbe1666550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:04.124921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e645ad9b-58be-4c30-9b14-e0dbe1666550 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:04.125574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e645ad9b-58be-4c30-9b14-e0dbe1666550 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:04.127847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e645ad9b-58be-4c30-9b14-e0dbe1666550 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:04.144233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 968/1945] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:04 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:04.145437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927bacb0-2002-443c-9728-6eeeeb705750 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:04.158462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927bacb0-2002-443c-9728-6eeeeb705750 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:04.164481 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e55f04-b902-4214-a304-dec342eb9f3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:04.165310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e55f04-b902-4214-a304-dec342eb9f3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:04.165590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e55f04-b902-4214-a304-dec342eb9f3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:04.165887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e55f04-b902-4214-a304-dec342eb9f3a admin admin] PATH_INFO: `/v3/projects/c61ce3bb85cd454a80fc3ba695875284/users/b0b78ee0114341738201d8fb6b583677/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:04.169599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e55f04-b902-4214-a304-dec342eb9f3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c61ce3bb85cd454a80fc3ba695875284, user_id=b0b78ee0114341738201d8fb6b583677, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:04.173428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e55f04-b902-4214-a304-dec342eb9f3a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:04.189767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 978/1946] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:04 2022] PUT /identity/v3/projects/c61ce3bb85cd454a80fc3ba695875284/users/b0b78ee0114341738201d8fb6b583677/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:04.200397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb953701-510f-452a-8759-97837fd46757 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:04.200687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb953701-510f-452a-8759-97837fd46757 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:04.200968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb953701-510f-452a-8759-97837fd46757 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:04.239699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bb953701-510f-452a-8759-97837fd46757 None None] MFA Rules not processed for user `b0b78ee0114341738201d8fb6b583677`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:04.241667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bb953701-510f-452a-8759-97837fd46757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:04.248529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e55f04-b902-4214-a304-dec342eb9f3a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:04.250378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-68e55f04-b902-4214-a304-dec342eb9f3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:04.388245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e55f04-b902-4214-a304-dec342eb9f3a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:04.388453 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 969/1947] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:04 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:04.396161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:04.396945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:04.397372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:04.397699 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] PATH_INFO: `/v3/projects/0aa60074f54b401e80b40aeeb4c42ead` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:04.399212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0aa60074f54b401e80b40aeeb4c42ead)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:04.401311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:04.401487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb953701-510f-452a-8759-97837fd46757 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:04.456823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0aa60074f54b401e80b40aeeb4c42ead', 'request_id': 'req-541291c4-5609-44f5-a0ae-ac364eae09e8'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:04.461525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb953701-510f-452a-8759-97837fd46757 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:04.463110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0aa60074f54b401e80b40aeeb4c42ead', 'request_id': 'req-541291c4-5609-44f5-a0ae-ac364eae09e8'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:04.469203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0aa60074f54b401e80b40aeeb4c42ead', 'request_id': 'req-541291c4-5609-44f5-a0ae-ac364eae09e8'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:04.482940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] The token cache is being invalidate because project 0aa60074f54b401e80b40aeeb4c42ead was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:04.483226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:04.483866 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:04.485409 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 979/1948] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:04 2022] DELETE /identity/v3/projects/0aa60074f54b401e80b40aeeb4c42ead => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:04.489233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:04.490524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:04.512442 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef3a13b-f5cd-42fb-939b-585d8ad93204 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:04.513081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef3a13b-f5cd-42fb-939b-585d8ad93204 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:04.513365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef3a13b-f5cd-42fb-939b-585d8ad93204 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:04.513633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef3a13b-f5cd-42fb-939b-585d8ad93204 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:04.515360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5ef3a13b-f5cd-42fb-939b-585d8ad93204 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:04.521929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:04.548318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:04.558949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541291c4-5609-44f5-a0ae-ac364eae09e8 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:04.563941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:04.564604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:04.564884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:04.565156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] PATH_INFO: `/v3/users/041232186cb14ae082e85f4e971f5ebb` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:04.566303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] RBAC: Authorizing `identity:delete_user(user_id=041232186cb14ae082e85f4e971f5ebb)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:04.568223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:04.582101 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef3a13b-f5cd-42fb-939b-585d8ad93204 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:04.583851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef3a13b-f5cd-42fb-939b-585d8ad93204 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:04.596315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 970/1949] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:37:04.623787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '041232186cb14ae082e85f4e971f5ebb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:04.632992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '041232186cb14ae082e85f4e971f5ebb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:04.633691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '041232186cb14ae082e85f4e971f5ebb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:04.634372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '041232186cb14ae082e85f4e971f5ebb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:04.649202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '041232186cb14ae082e85f4e971f5ebb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:04.663335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 980/1950] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:04 2022] DELETE /identity/v3/users/041232186cb14ae082e85f4e971f5ebb => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:05.155905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef3a13b-f5cd-42fb-939b-585d8ad93204 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:05.171208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef3a13b-f5cd-42fb-939b-585d8ad93204 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:05.240537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:05.241393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:05.241675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:05.241979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] PATH_INFO: `/v3/projects/2dee5286848e4dff8ec448685c2ecf3d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:05.243121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2dee5286848e4dff8ec448685c2ecf3d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:05.245098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:05.295256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2dee5286848e4dff8ec448685c2ecf3d', 'request_id': 'req-a06a1bd3-7c25-42af-840b-dedc320a5f44'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.302512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2dee5286848e4dff8ec448685c2ecf3d', 'request_id': 'req-a06a1bd3-7c25-42af-840b-dedc320a5f44'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.311883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2dee5286848e4dff8ec448685c2ecf3d', 'request_id': 'req-a06a1bd3-7c25-42af-840b-dedc320a5f44'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.324715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] The token cache is being invalidate because project 2dee5286848e4dff8ec448685c2ecf3d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:05.325032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.325681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.327188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 971/1951] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:05 2022] DELETE /identity/v3/projects/2dee5286848e4dff8ec448685c2ecf3d => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:05.331204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:05.332821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:05.346381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed29b5bc-3581-4638-bdc0-d64365b5a671 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:05.404896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:05.405532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:05.405801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:05.406072 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] PATH_INFO: `/v3/users/4ab59ff0a7bf41febccc7a7cb7be2651` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:05.407191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=4ab59ff0a7bf41febccc7a7cb7be2651)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:05.409139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:05.457722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4ab59ff0a7bf41febccc7a7cb7be2651'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.459343 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:05.466606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4ab59ff0a7bf41febccc7a7cb7be2651'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.467265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4ab59ff0a7bf41febccc7a7cb7be2651'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.467976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4ab59ff0a7bf41febccc7a7cb7be2651'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.479962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-d391e041-f806-4c69-bdd1-2fb72418de1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4ab59ff0a7bf41febccc7a7cb7be2651'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.492921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 981/1952] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:05 2022] DELETE /identity/v3/users/4ab59ff0a7bf41febccc7a7cb7be2651 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:05.761567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:05.763337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:05.777219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06a1bd3-7c25-42af-840b-dedc320a5f44 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:05.836529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:05.837179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:05.837458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:05.837732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] PATH_INFO: `/v3/projects/e211e437990541949d4994559f1f323a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:05.838843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e211e437990541949d4994559f1f323a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:05.840795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:05.843018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:05.893946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e211e437990541949d4994559f1f323a', 'request_id': 'req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.901346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e211e437990541949d4994559f1f323a', 'request_id': 'req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.910660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e211e437990541949d4994559f1f323a', 'request_id': 'req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.922403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] The token cache is being invalidate because project e211e437990541949d4994559f1f323a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:05.922694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.923361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-81c93bbf-d405-4cc5-b2ec-1dcb87af9f77 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:05.925100 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 972/1953] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:05 2022] DELETE /identity/v3/projects/e211e437990541949d4994559f1f323a => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:05.933116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43660f26-9e5d-4033-9e9f-5eba7986770b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:05.933531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43660f26-9e5d-4033-9e9f-5eba7986770b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:05.933746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43660f26-9e5d-4033-9e9f-5eba7986770b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:05.972217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-43660f26-9e5d-4033-9e9f-5eba7986770b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:05.973676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-43660f26-9e5d-4033-9e9f-5eba7986770b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:06.061711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 982/1954] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:06.072515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697386c6-5cb9-447d-9ce8-1cbe3ade1774 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:06.072840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697386c6-5cb9-447d-9ce8-1cbe3ade1774 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:06.073376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697386c6-5cb9-447d-9ce8-1cbe3ade1774 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:06.102991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-697386c6-5cb9-447d-9ce8-1cbe3ade1774 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:06.104402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-697386c6-5cb9-447d-9ce8-1cbe3ade1774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:06.121108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43660f26-9e5d-4033-9e9f-5eba7986770b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:06.128720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 973/1955] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:06.134263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43660f26-9e5d-4033-9e9f-5eba7986770b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:06.142381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5b81d2-4a17-4b7b-a425-88fdfba71fb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:06.143264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5b81d2-4a17-4b7b-a425-88fdfba71fb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:06.143694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5b81d2-4a17-4b7b-a425-88fdfba71fb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:06.144128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5b81d2-4a17-4b7b-a425-88fdfba71fb1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:06.144836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5b81d2-4a17-4b7b-a425-88fdfba71fb1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:06.145533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5b81d2-4a17-4b7b-a425-88fdfba71fb1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:06.146825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5b81d2-4a17-4b7b-a425-88fdfba71fb1 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:06.165190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 983/1956] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:06.166571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697386c6-5cb9-447d-9ce8-1cbe3ade1774 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:06.176749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697386c6-5cb9-447d-9ce8-1cbe3ade1774 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:06.182441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5fe607-c3eb-418d-981e-0bf86abe2a74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:06.183374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5fe607-c3eb-418d-981e-0bf86abe2a74 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:06.183719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5fe607-c3eb-418d-981e-0bf86abe2a74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:06.184093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5fe607-c3eb-418d-981e-0bf86abe2a74 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:06.185414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5fe607-c3eb-418d-981e-0bf86abe2a74 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:06.187342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5fe607-c3eb-418d-981e-0bf86abe2a74 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:06.220912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 974/1957] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:06 2022] POST /identity/v3/projects => generated 377 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:06.225102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5b81d2-4a17-4b7b-a425-88fdfba71fb1 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:06.236125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5b81d2-4a17-4b7b-a425-88fdfba71fb1 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:06.312702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae6366c-81c8-4fa5-a31b-7223e2270236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:06.313306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae6366c-81c8-4fa5-a31b-7223e2270236 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:06.313590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae6366c-81c8-4fa5-a31b-7223e2270236 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:06.313864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae6366c-81c8-4fa5-a31b-7223e2270236 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:06.314883 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae6366c-81c8-4fa5-a31b-7223e2270236 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:06.317108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae6366c-81c8-4fa5-a31b-7223e2270236 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:06.344798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 984/1958] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:06 2022] POST /identity/v3/users => generated 289 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:06.345927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5fe607-c3eb-418d-981e-0bf86abe2a74 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:06.359393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5fe607-c3eb-418d-981e-0bf86abe2a74 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:06.365713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a66d4b-c4c1-4f02-ac81-dbd6341be4c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:06.367698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a66d4b-c4c1-4f02-ac81-dbd6341be4c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:06.368142 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a66d4b-c4c1-4f02-ac81-dbd6341be4c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:06.368530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a66d4b-c4c1-4f02-ac81-dbd6341be4c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:06.369428 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a66d4b-c4c1-4f02-ac81-dbd6341be4c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:06.371171 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a66d4b-c4c1-4f02-ac81-dbd6341be4c4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:06.383879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 975/1959] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:06 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:06.389170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae6366c-81c8-4fa5-a31b-7223e2270236 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:06.405826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae6366c-81c8-4fa5-a31b-7223e2270236 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:06.424604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202b239a-657e-49d3-8e75-70675176995b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:06.426424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202b239a-657e-49d3-8e75-70675176995b admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:06.426715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202b239a-657e-49d3-8e75-70675176995b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:06.426993 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202b239a-657e-49d3-8e75-70675176995b admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:06.427951 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202b239a-657e-49d3-8e75-70675176995b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:06.434281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202b239a-657e-49d3-8e75-70675176995b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:06.455833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 985/1960] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:06 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:06.457170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a66d4b-c4c1-4f02-ac81-dbd6341be4c4 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:06.469054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a66d4b-c4c1-4f02-ac81-dbd6341be4c4 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:06.474681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8014e4e2-76df-4f10-a528-181af269ebd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:06.476501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8014e4e2-76df-4f10-a528-181af269ebd7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:06.477107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8014e4e2-76df-4f10-a528-181af269ebd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:06.477563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8014e4e2-76df-4f10-a528-181af269ebd7 admin admin] PATH_INFO: `/v3/projects/686b1cfff48f4fc8a140d2e2c267a3c9/users/52c53fe46346489dbf27c55747ae5291/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:06.506299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8014e4e2-76df-4f10-a528-181af269ebd7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=686b1cfff48f4fc8a140d2e2c267a3c9, user_id=52c53fe46346489dbf27c55747ae5291, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:06.511323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8014e4e2-76df-4f10-a528-181af269ebd7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:06.549702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 976/1961] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:06 2022] PUT /identity/v3/projects/686b1cfff48f4fc8a140d2e2c267a3c9/users/52c53fe46346489dbf27c55747ae5291/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:06.561882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeffd41-cf1f-40b1-9cb4-d85b75ad1eea None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:06.562890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeffd41-cf1f-40b1-9cb4-d85b75ad1eea None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:06.563170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeffd41-cf1f-40b1-9cb4-d85b75ad1eea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:06.605967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-dbeffd41-cf1f-40b1-9cb4-d85b75ad1eea None None] MFA Rules not processed for user `52c53fe46346489dbf27c55747ae5291`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:06.607481 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-dbeffd41-cf1f-40b1-9cb4-d85b75ad1eea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:06.636157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:06.717848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 986/1962] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:06 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:06.721689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8014e4e2-76df-4f10-a528-181af269ebd7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:06.835853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8014e4e2-76df-4f10-a528-181af269ebd7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:06.845198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc2f6e5-5d3d-4e08-9925-7a4d817560ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:06.845818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc2f6e5-5d3d-4e08-9925-7a4d817560ac admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:06.846303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc2f6e5-5d3d-4e08-9925-7a4d817560ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:06.846585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc2f6e5-5d3d-4e08-9925-7a4d817560ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:06.847411 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc2f6e5-5d3d-4e08-9925-7a4d817560ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:06.849956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc2f6e5-5d3d-4e08-9925-7a4d817560ac admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:06.886050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 977/1963] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:06 2022] POST /identity/v3/projects => generated 377 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:06.891129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeffd41-cf1f-40b1-9cb4-d85b75ad1eea None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:06.908873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeffd41-cf1f-40b1-9cb4-d85b75ad1eea None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:06.966100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810211a9-4ad6-4389-a59b-1ddf801e3617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:06.966705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810211a9-4ad6-4389-a59b-1ddf801e3617 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:06.966979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810211a9-4ad6-4389-a59b-1ddf801e3617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:06.967250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810211a9-4ad6-4389-a59b-1ddf801e3617 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:06.967903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810211a9-4ad6-4389-a59b-1ddf801e3617 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:06.969810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810211a9-4ad6-4389-a59b-1ddf801e3617 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:06.990419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 987/1964] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:06 2022] POST /identity/v3/users => generated 289 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:06.994873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc2f6e5-5d3d-4e08-9925-7a4d817560ac admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.005091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc2f6e5-5d3d-4e08-9925-7a4d817560ac admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.010246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd04cbe-f559-4387-a397-c41b89fcf9f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.010837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd04cbe-f559-4387-a397-c41b89fcf9f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.011119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd04cbe-f559-4387-a397-c41b89fcf9f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.011398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd04cbe-f559-4387-a397-c41b89fcf9f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.012039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd04cbe-f559-4387-a397-c41b89fcf9f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.013687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd04cbe-f559-4387-a397-c41b89fcf9f7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.025362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 978/1965] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:07.029284 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810211a9-4ad6-4389-a59b-1ddf801e3617 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.041184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810211a9-4ad6-4389-a59b-1ddf801e3617 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.047278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab8f9f5-f93f-46bb-87c0-1c237c9839a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.047721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab8f9f5-f93f-46bb-87c0-1c237c9839a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.048007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab8f9f5-f93f-46bb-87c0-1c237c9839a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.048283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab8f9f5-f93f-46bb-87c0-1c237c9839a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.048972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab8f9f5-f93f-46bb-87c0-1c237c9839a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.050576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab8f9f5-f93f-46bb-87c0-1c237c9839a6 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.064535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 988/1966] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:07 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:07.065892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd04cbe-f559-4387-a397-c41b89fcf9f7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.071764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:07.078324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd04cbe-f559-4387-a397-c41b89fcf9f7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.083958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729330c-3b38-41be-a8d6-354c16ce9877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.084692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729330c-3b38-41be-a8d6-354c16ce9877 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.084965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729330c-3b38-41be-a8d6-354c16ce9877 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.085247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729330c-3b38-41be-a8d6-354c16ce9877 admin admin] PATH_INFO: `/v3/projects/84f68256299e48a781f98428594e9aba/users/6775db4bcd714da6918383f57aa66804/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.100429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729330c-3b38-41be-a8d6-354c16ce9877 admin admin] RBAC: Authorizing `identity:create_grant(project_id=84f68256299e48a781f98428594e9aba, user_id=6775db4bcd714da6918383f57aa66804, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.103310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729330c-3b38-41be-a8d6-354c16ce9877 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.117562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 979/1967] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:07 2022] PUT /identity/v3/projects/84f68256299e48a781f98428594e9aba/users/6775db4bcd714da6918383f57aa66804/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:07.132908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a9f594-5be0-4fce-8a2e-4f86e09f72cc None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.132908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a9f594-5be0-4fce-8a2e-4f86e09f72cc None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.132908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a9f594-5be0-4fce-8a2e-4f86e09f72cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.162819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-a1a9f594-5be0-4fce-8a2e-4f86e09f72cc None None] MFA Rules not processed for user `6775db4bcd714da6918383f57aa66804`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:07.164077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a1a9f594-5be0-4fce-8a2e-4f86e09f72cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:07.237441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 989/1968] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:07.242327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729330c-3b38-41be-a8d6-354c16ce9877 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.248368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a9f594-5be0-4fce-8a2e-4f86e09f72cc None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.249726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a1a9f594-5be0-4fce-8a2e-4f86e09f72cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:07.281340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:07.305844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729330c-3b38-41be-a8d6-354c16ce9877 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.307594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a9f594-5be0-4fce-8a2e-4f86e09f72cc None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.310369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07ad91d-d36e-4353-a163-8ba735f724bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.310964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ad91d-d36e-4353-a163-8ba735f724bd admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.311143 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaac34e-c921-42cc-9dc1-e7c3bc7d3a7b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.311287 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ad91d-d36e-4353-a163-8ba735f724bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.311498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ad91d-d36e-4353-a163-8ba735f724bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.311741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaac34e-c921-42cc-9dc1-e7c3bc7d3a7b service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.312020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaac34e-c921-42cc-9dc1-e7c3bc7d3a7b service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.312250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07ad91d-d36e-4353-a163-8ba735f724bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.312358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaac34e-c921-42cc-9dc1-e7c3bc7d3a7b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.314026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-faaac34e-c921-42cc-9dc1-e7c3bc7d3a7b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:07.314183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07ad91d-d36e-4353-a163-8ba735f724bd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.345935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 980/1969] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:07 2022] POST /identity/v3/projects => generated 377 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:07.346916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07ad91d-d36e-4353-a163-8ba735f724bd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.375857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaac34e-c921-42cc-9dc1-e7c3bc7d3a7b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.377494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaac34e-c921-42cc-9dc1-e7c3bc7d3a7b service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.386886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 990/1970] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:37:07.413419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07ad91d-d36e-4353-a163-8ba735f724bd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.418293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42315e46-71d8-4862-909d-47682a6dcd25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.419036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42315e46-71d8-4862-909d-47682a6dcd25 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.420389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42315e46-71d8-4862-909d-47682a6dcd25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.420389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42315e46-71d8-4862-909d-47682a6dcd25 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.420389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42315e46-71d8-4862-909d-47682a6dcd25 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.422282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42315e46-71d8-4862-909d-47682a6dcd25 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.449130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 981/1971] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:07 2022] POST /identity/v3/users => generated 295 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:07.453223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaac34e-c921-42cc-9dc1-e7c3bc7d3a7b service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.464938 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaac34e-c921-42cc-9dc1-e7c3bc7d3a7b service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.469626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cec2ec-7f1e-460f-bdee-6558ebd6cd0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.470248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cec2ec-7f1e-460f-bdee-6558ebd6cd0f admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.470530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cec2ec-7f1e-460f-bdee-6558ebd6cd0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.470802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cec2ec-7f1e-460f-bdee-6558ebd6cd0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.471462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cec2ec-7f1e-460f-bdee-6558ebd6cd0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.473109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cec2ec-7f1e-460f-bdee-6558ebd6cd0f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.489682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42315e46-71d8-4862-909d-47682a6dcd25 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.490094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 991/1972] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:07 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:07.500632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42315e46-71d8-4862-909d-47682a6dcd25 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.505521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134b4379-241e-4a21-bc62-0d4a111eee6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.506266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134b4379-241e-4a21-bc62-0d4a111eee6c admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.506711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134b4379-241e-4a21-bc62-0d4a111eee6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.506901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134b4379-241e-4a21-bc62-0d4a111eee6c admin admin] PATH_INFO: `/v3/projects/474d52be0f6845dbb6043b564d3bc17d/users/2d78eb25fd73444abeaaab742778c7f5/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.523311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134b4379-241e-4a21-bc62-0d4a111eee6c admin admin] RBAC: Authorizing `identity:create_grant(project_id=474d52be0f6845dbb6043b564d3bc17d, user_id=2d78eb25fd73444abeaaab742778c7f5, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.526285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134b4379-241e-4a21-bc62-0d4a111eee6c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.543352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 982/1973] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:07 2022] PUT /identity/v3/projects/474d52be0f6845dbb6043b564d3bc17d/users/2d78eb25fd73444abeaaab742778c7f5/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:07.547601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cec2ec-7f1e-460f-bdee-6558ebd6cd0f admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.559932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cec2ec-7f1e-460f-bdee-6558ebd6cd0f admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.630353 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0834439-8a60-4e42-a7db-8e0b448db23b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.631018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0834439-8a60-4e42-a7db-8e0b448db23b admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.631745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0834439-8a60-4e42-a7db-8e0b448db23b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.631745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0834439-8a60-4e42-a7db-8e0b448db23b admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.632984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0834439-8a60-4e42-a7db-8e0b448db23b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.635195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0834439-8a60-4e42-a7db-8e0b448db23b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.649311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 992/1974] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:07 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:07.653379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134b4379-241e-4a21-bc62-0d4a111eee6c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.667052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134b4379-241e-4a21-bc62-0d4a111eee6c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.672061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49864817-6d9e-48f1-a544-6ddb3ea26b3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.672819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49864817-6d9e-48f1-a544-6ddb3ea26b3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.673102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49864817-6d9e-48f1-a544-6ddb3ea26b3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.674117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49864817-6d9e-48f1-a544-6ddb3ea26b3c admin admin] PATH_INFO: `/v3/projects/474d52be0f6845dbb6043b564d3bc17d/users/2d78eb25fd73444abeaaab742778c7f5/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.676275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49864817-6d9e-48f1-a544-6ddb3ea26b3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=474d52be0f6845dbb6043b564d3bc17d, user_id=2d78eb25fd73444abeaaab742778c7f5, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.679607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49864817-6d9e-48f1-a544-6ddb3ea26b3c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.692318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 983/1975] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:07 2022] PUT /identity/v3/projects/474d52be0f6845dbb6043b564d3bc17d/users/2d78eb25fd73444abeaaab742778c7f5/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:07.703278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e89f6-9c61-4351-9751-76a075cadb84 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.703817 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e89f6-9c61-4351-9751-76a075cadb84 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.704388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e89f6-9c61-4351-9751-76a075cadb84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.742417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-9a7e89f6-9c61-4351-9751-76a075cadb84 None None] MFA Rules not processed for user `2d78eb25fd73444abeaaab742778c7f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:07.743658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9a7e89f6-9c61-4351-9751-76a075cadb84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:07.815265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:07.827865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 993/1976] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:07 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:07.839476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49864817-6d9e-48f1-a544-6ddb3ea26b3c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:07.841244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-49864817-6d9e-48f1-a544-6ddb3ea26b3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:07.880912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:07.949736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49864817-6d9e-48f1-a544-6ddb3ea26b3c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:07.954821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858ecfd6-c5d2-4650-9c90-f26041e3e08b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:07.954821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858ecfd6-c5d2-4650-9c90-f26041e3e08b service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:07.955182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858ecfd6-c5d2-4650-9c90-f26041e3e08b service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:07.955501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858ecfd6-c5d2-4650-9c90-f26041e3e08b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:07.966259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858ecfd6-c5d2-4650-9c90-f26041e3e08b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:07.968513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858ecfd6-c5d2-4650-9c90-f26041e3e08b service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:07.984714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 984/1977] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:07 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:08.051571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e89f6-9c61-4351-9751-76a075cadb84 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:08.063446 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e89f6-9c61-4351-9751-76a075cadb84 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:08.067091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea23f4c2-1f3e-4d3b-84e7-e897f316d9a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:08.067707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea23f4c2-1f3e-4d3b-84e7-e897f316d9a3 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:08.067981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea23f4c2-1f3e-4d3b-84e7-e897f316d9a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:08.068256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea23f4c2-1f3e-4d3b-84e7-e897f316d9a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:08.158051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea23f4c2-1f3e-4d3b-84e7-e897f316d9a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:08.159594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea23f4c2-1f3e-4d3b-84e7-e897f316d9a3 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:08.169543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 994/1978] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:08.241624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858ecfd6-c5d2-4650-9c90-f26041e3e08b service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:08.250333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858ecfd6-c5d2-4650-9c90-f26041e3e08b service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:08.254904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c30637-48d8-40cb-8ed3-e71a61f96225 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:08.255509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c30637-48d8-40cb-8ed3-e71a61f96225 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:08.255829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c30637-48d8-40cb-8ed3-e71a61f96225 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:08.256109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c30637-48d8-40cb-8ed3-e71a61f96225 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:08.257941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-28c30637-48d8-40cb-8ed3-e71a61f96225 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:08.329543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c30637-48d8-40cb-8ed3-e71a61f96225 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:08.331121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c30637-48d8-40cb-8ed3-e71a61f96225 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:08.341848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 985/1979] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:08 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:11.360751 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea23f4c2-1f3e-4d3b-84e7-e897f316d9a3 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:11.362248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ea23f4c2-1f3e-4d3b-84e7-e897f316d9a3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:11.531543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea23f4c2-1f3e-4d3b-84e7-e897f316d9a3 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:11.539120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:11.540682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:11.541070 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:11.541665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] PATH_INFO: `/v3/users/13b7ef0b77114af6b5d1d162a5e9434f` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:11.542836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=13b7ef0b77114af6b5d1d162a5e9434f)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:11.545346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:11.671558 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13b7ef0b77114af6b5d1d162a5e9434f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:11.685036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13b7ef0b77114af6b5d1d162a5e9434f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:11.685854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13b7ef0b77114af6b5d1d162a5e9434f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:11.686662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13b7ef0b77114af6b5d1d162a5e9434f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:11.709059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13b7ef0b77114af6b5d1d162a5e9434f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:11.736627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 995/1980] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:11 2022] DELETE /identity/v3/users/13b7ef0b77114af6b5d1d162a5e9434f => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:11.741206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627301c1-4b6f-402a-ae1f-90163b214648 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:11.742268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627301c1-4b6f-402a-ae1f-90163b214648 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:11.742845 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627301c1-4b6f-402a-ae1f-90163b214648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:11.780062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-627301c1-4b6f-402a-ae1f-90163b214648 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:11.782475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-627301c1-4b6f-402a-ae1f-90163b214648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:11.860200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 986/1981] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:11.882767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:11.884807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:11.903501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:11.962793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617ce5df-f2ca-4ee1-b1a4-304e83e0a2b8 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:11.972917 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28e0ea6-c41b-4475-9a9c-79941cc2d134 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:11.973655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e0ea6-c41b-4475-9a9c-79941cc2d134 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:11.974035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e0ea6-c41b-4475-9a9c-79941cc2d134 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:11.974409 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e0ea6-c41b-4475-9a9c-79941cc2d134 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:12.003500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28e0ea6-c41b-4475-9a9c-79941cc2d134 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:12.005107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28e0ea6-c41b-4475-9a9c-79941cc2d134 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:12.019058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 996/1982] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:12.467900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627301c1-4b6f-402a-ae1f-90163b214648 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:12.469536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-627301c1-4b6f-402a-ae1f-90163b214648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:12.494527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627301c1-4b6f-402a-ae1f-90163b214648 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:12.501421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:12.502271 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:12.502561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:12.502840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] PATH_INFO: `/v3/users/52c53fe46346489dbf27c55747ae5291` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:12.504979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=52c53fe46346489dbf27c55747ae5291)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:12.506709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:12.566312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52c53fe46346489dbf27c55747ae5291'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.570517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:12.582076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52c53fe46346489dbf27c55747ae5291'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.583239 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52c53fe46346489dbf27c55747ae5291'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.586234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52c53fe46346489dbf27c55747ae5291'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.597699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52c53fe46346489dbf27c55747ae5291'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.621441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 987/1983] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:12 2022] DELETE /identity/v3/users/52c53fe46346489dbf27c55747ae5291 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:12.626006 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd972c9c-ce40-4153-8c9f-b703ccb0ec06 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:12.627180 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd972c9c-ce40-4153-8c9f-b703ccb0ec06 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:12.627682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd972c9c-ce40-4153-8c9f-b703ccb0ec06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:12.654958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-bd972c9c-ce40-4153-8c9f-b703ccb0ec06 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:12.656316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-bd972c9c-ce40-4153-8c9f-b703ccb0ec06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:12.671169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:12.673433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:12.806159 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce283bb-3365-4443-9452-9a27d33469f0 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:12.813134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 997/1984] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:12.814202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd972c9c-ce40-4153-8c9f-b703ccb0ec06 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:12.817059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:12.817716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-bd972c9c-ce40-4153-8c9f-b703ccb0ec06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:12.817839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:12.818731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:12.819404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] PATH_INFO: `/v3/projects/b4e2ab570b2c461f9ad8fc19dcbe7f0f` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:12.820784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b4e2ab570b2c461f9ad8fc19dcbe7f0f)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:12.822784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:12.842650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd972c9c-ce40-4153-8c9f-b703ccb0ec06 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:12.854021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:12.855083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:12.855371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:12.855651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] PATH_INFO: `/v3/users/ba818bad3e4948a1820ef8f825ac4078` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:12.857686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba818bad3e4948a1820ef8f825ac4078)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:12.859729 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:12.882608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b4e2ab570b2c461f9ad8fc19dcbe7f0f', 'request_id': 'req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.891085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b4e2ab570b2c461f9ad8fc19dcbe7f0f', 'request_id': 'req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.902690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b4e2ab570b2c461f9ad8fc19dcbe7f0f', 'request_id': 'req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.920402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] The token cache is being invalidate because project b4e2ab570b2c461f9ad8fc19dcbe7f0f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:12.920804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.921057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba818bad3e4948a1820ef8f825ac4078'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.921483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.923118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 988/1985] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:12 2022] DELETE /identity/v3/projects/b4e2ab570b2c461f9ad8fc19dcbe7f0f => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:12.924430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:12.927253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:12.935937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba818bad3e4948a1820ef8f825ac4078'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.936851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba818bad3e4948a1820ef8f825ac4078'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.937498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba818bad3e4948a1820ef8f825ac4078'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.953565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba818bad3e4948a1820ef8f825ac4078'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:12.970964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 998/1986] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:12 2022] DELETE /identity/v3/users/ba818bad3e4948a1820ef8f825ac4078 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:12.971943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:12.973401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:13.020217 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff43d9f6-0cfc-401b-a756-952b6c0d8cb3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:13.031351 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:13.033026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:INSERT=2,keystone:DELETE=10,keystone:SELECT=88 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:13.131694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c754a3a-2562-4927-bb1b-5720757e4539 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:13.134998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519bd6df-65da-41b1-804d-1c9640725012 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:13.137126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bd6df-65da-41b1-804d-1c9640725012 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:13.138490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bd6df-65da-41b1-804d-1c9640725012 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:13.138776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bd6df-65da-41b1-804d-1c9640725012 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:13.141973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-519bd6df-65da-41b1-804d-1c9640725012 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:13.146241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:13.147868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:13.148438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:13.148933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] PATH_INFO: `/v3/users/ff49582d0f2747eab23ec79b63640e4b` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:13.151253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff49582d0f2747eab23ec79b63640e4b)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:13.154594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:13.159161 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519bd6df-65da-41b1-804d-1c9640725012 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:13.160859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519bd6df-65da-41b1-804d-1c9640725012 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:13.188053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 989/1987] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:13.190438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b22f8c-78de-48c8-9f62-a1d22b7eec1d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:13.190438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b22f8c-78de-48c8-9f62-a1d22b7eec1d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:13.190692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b22f8c-78de-48c8-9f62-a1d22b7eec1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:13.217409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b0b22f8c-78de-48c8-9f62-a1d22b7eec1d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:13.221873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b0b22f8c-78de-48c8-9f62-a1d22b7eec1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:13.236233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 990/1988] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:13.239398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff49582d0f2747eab23ec79b63640e4b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.250444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff49582d0f2747eab23ec79b63640e4b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.251136 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff49582d0f2747eab23ec79b63640e4b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.251930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff49582d0f2747eab23ec79b63640e4b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.265864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b22f8c-78de-48c8-9f62-a1d22b7eec1d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:13.268967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b0b22f8c-78de-48c8-9f62-a1d22b7eec1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:13.274289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff49582d0f2747eab23ec79b63640e4b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.295761 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b22f8c-78de-48c8-9f62-a1d22b7eec1d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:13.298868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 999/1989] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:12 2022] DELETE /identity/v3/users/ff49582d0f2747eab23ec79b63640e4b => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:13.378810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e5fb0e-6015-4c21-ac90-5326f42254d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:13.379702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e5fb0e-6015-4c21-ac90-5326f42254d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:13.379993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e5fb0e-6015-4c21-ac90-5326f42254d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:13.380415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e5fb0e-6015-4c21-ac90-5326f42254d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:13.383805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e8e5fb0e-6015-4c21-ac90-5326f42254d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:13.505378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e5fb0e-6015-4c21-ac90-5326f42254d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:13.511062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e5fb0e-6015-4c21-ac90-5326f42254d6 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:13.557670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 991/1990] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:13.589012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:13.591244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:13.618265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac92b3e-a4ed-4301-b220-5324e7c7d405 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:13.627644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e5fb0e-6015-4c21-ac90-5326f42254d6 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:13.629828 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e8e5fb0e-6015-4c21-ac90-5326f42254d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:13.658717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e5fb0e-6015-4c21-ac90-5326f42254d6 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:13.659463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:13.660118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:13.660432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:13.660703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] PATH_INFO: `/v3/projects/686b1cfff48f4fc8a140d2e2c267a3c9` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:13.662067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=686b1cfff48f4fc8a140d2e2c267a3c9)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:13.665101 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:13.665984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:13.666796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:13.667080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:13.667358 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] PATH_INFO: `/v3/users/98ab328072ee4f85ac3835bc920b01ff` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:13.668590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] RBAC: Authorizing `identity:delete_user(user_id=98ab328072ee4f85ac3835bc920b01ff)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:13.670540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:13.731898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98ab328072ee4f85ac3835bc920b01ff'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.733323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '686b1cfff48f4fc8a140d2e2c267a3c9', 'request_id': 'req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.738391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:13.745507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '686b1cfff48f4fc8a140d2e2c267a3c9', 'request_id': 'req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.746382 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98ab328072ee4f85ac3835bc920b01ff'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.748495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98ab328072ee4f85ac3835bc920b01ff'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.749623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98ab328072ee4f85ac3835bc920b01ff'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.758951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c9c7ea91-8cb4-4cf8-894c-857a3fcef850 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98ab328072ee4f85ac3835bc920b01ff'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.759731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '686b1cfff48f4fc8a140d2e2c267a3c9', 'request_id': 'req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.788048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] The token cache is being invalidate because project 686b1cfff48f4fc8a140d2e2c267a3c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:13.789333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.789599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 992/1991] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:13 2022] DELETE /identity/v3/users/98ab328072ee4f85ac3835bc920b01ff => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:13.790439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.792132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1000/1992] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:13 2022] DELETE /identity/v3/projects/686b1cfff48f4fc8a140d2e2c267a3c9 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:13.794183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f2b08e-3672-448c-ade1-0cb9697a335a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:13.794802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f2b08e-3672-448c-ade1-0cb9697a335a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:13.795502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f2b08e-3672-448c-ade1-0cb9697a335a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:13.796103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:13.797521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:13.821896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-56f2b08e-3672-448c-ade1-0cb9697a335a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:13.825026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-56f2b08e-3672-448c-ade1-0cb9697a335a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:13.840375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:13.889225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f8af56-42b4-44bc-a8e8-9b87510b6d2b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:13.895722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 993/1993] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:13.907831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f2b08e-3672-448c-ade1-0cb9697a335a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:13.908011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:13.908712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:13.909000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:13.909283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] PATH_INFO: `/v3/users/6775db4bcd714da6918383f57aa66804` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:13.910863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-56f2b08e-3672-448c-ade1-0cb9697a335a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:13.911328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6775db4bcd714da6918383f57aa66804)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:13.914060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:13.984955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6775db4bcd714da6918383f57aa66804'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.989562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f2b08e-3672-448c-ade1-0cb9697a335a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:13.995368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6775db4bcd714da6918383f57aa66804'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.996054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6775db4bcd714da6918383f57aa66804'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:13.996734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6775db4bcd714da6918383f57aa66804'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:14.009212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597096e0-3762-4706-8422-197644770f4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:14.009837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597096e0-3762-4706-8422-197644770f4f service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:14.011780 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597096e0-3762-4706-8422-197644770f4f service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:14.012072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597096e0-3762-4706-8422-197644770f4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:14.015046 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6775db4bcd714da6918383f57aa66804'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:14.018398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-597096e0-3762-4706-8422-197644770f4f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:14.042257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1001/1994] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:13 2022] DELETE /identity/v3/users/6775db4bcd714da6918383f57aa66804 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:14.043802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597096e0-3762-4706-8422-197644770f4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:14.051804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597096e0-3762-4706-8422-197644770f4f service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:14.066016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 994/1995] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:37:14.681560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:14.683273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:14.815975 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e042fea-834c-4b29-a38a-1e8aa90fc580 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:14.830136 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:14.831319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:14.831728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:14.832276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] PATH_INFO: `/v3/projects/a4ec12eace284a28a8cf620f04718707` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:14.833497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4ec12eace284a28a8cf620f04718707)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:14.839608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:14.943238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4ec12eace284a28a8cf620f04718707', 'request_id': 'req-69b11201-70f7-4226-b67e-80c2af479d9c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:14.948276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:14.964122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597096e0-3762-4706-8422-197644770f4f service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:14.966785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-597096e0-3762-4706-8422-197644770f4f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:14.966916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4ec12eace284a28a8cf620f04718707', 'request_id': 'req-69b11201-70f7-4226-b67e-80c2af479d9c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:14.982716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4ec12eace284a28a8cf620f04718707', 'request_id': 'req-69b11201-70f7-4226-b67e-80c2af479d9c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:14.991503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597096e0-3762-4706-8422-197644770f4f service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:15.008187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] The token cache is being invalidate because project a4ec12eace284a28a8cf620f04718707 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:15.008489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.010870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.014744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1002/1996] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:14 2022] DELETE /identity/v3/projects/a4ec12eace284a28a8cf620f04718707 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:15.019395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:15.023359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:15.136531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:15.137244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.137521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.137976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] PATH_INFO: `/v3/projects/84f68256299e48a781f98428594e9aba` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.139640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] RBAC: Authorizing `identity:delete_project(project_id=84f68256299e48a781f98428594e9aba)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:15.143523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:15.175800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b11201-70f7-4226-b67e-80c2af479d9c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:15.185992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:15.187199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:15.188765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.189058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.189330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] PATH_INFO: `/v3/projects/0e5203ad82e245e88d5d638c60b02d49` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.192923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e5203ad82e245e88d5d638c60b02d49)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:15.195525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:15.249745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84f68256299e48a781f98428594e9aba', 'request_id': 'req-e216e137-81c4-43d6-9848-88702245f24c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.265206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84f68256299e48a781f98428594e9aba', 'request_id': 'req-e216e137-81c4-43d6-9848-88702245f24c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.268632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e5203ad82e245e88d5d638c60b02d49', 'request_id': 'req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.276560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e5203ad82e245e88d5d638c60b02d49', 'request_id': 'req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.278425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84f68256299e48a781f98428594e9aba', 'request_id': 'req-e216e137-81c4-43d6-9848-88702245f24c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.285045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e5203ad82e245e88d5d638c60b02d49', 'request_id': 'req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.296441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] The token cache is being invalidate because project 84f68256299e48a781f98428594e9aba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:15.297164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.298617 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e216e137-81c4-43d6-9848-88702245f24c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.300912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 995/1997] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:14 2022] DELETE /identity/v3/projects/84f68256299e48a781f98428594e9aba => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:15.302626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f8059e-cfd6-4576-b35a-dc0c968faf48 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.303469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f8059e-cfd6-4576-b35a-dc0c968faf48 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.304724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f8059e-cfd6-4576-b35a-dc0c968faf48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.304995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] The token cache is being invalidate because project 0e5203ad82e245e88d5d638c60b02d49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:15.305444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.306579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.311092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1003/1998] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:15 2022] DELETE /identity/v3/projects/0e5203ad82e245e88d5d638c60b02d49 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:15.312752 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:15.314392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:15.338249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-76f8059e-cfd6-4576-b35a-dc0c968faf48 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:15.339497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-76f8059e-cfd6-4576-b35a-dc0c968faf48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:15.406047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179e98cb-59dc-4ce3-8f2b-801e1d5af1ae admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:15.416519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:15.417190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.417506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.417747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] PATH_INFO: `/v3/projects/01a95b064cf4490bab78eb2fa2fd026d` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.419068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01a95b064cf4490bab78eb2fa2fd026d)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:15.420090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 996/1999] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:15.421201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:15.427180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f8059e-cfd6-4576-b35a-dc0c968faf48 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:15.428686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-76f8059e-cfd6-4576-b35a-dc0c968faf48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:15.434601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:15.442984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f8059e-cfd6-4576-b35a-dc0c968faf48 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:15.448562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:15.449478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.449618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.449988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] PATH_INFO: `/v3/users/2d78eb25fd73444abeaaab742778c7f5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.451174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d78eb25fd73444abeaaab742778c7f5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:15.453212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:15.483720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01a95b064cf4490bab78eb2fa2fd026d', 'request_id': 'req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.492749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01a95b064cf4490bab78eb2fa2fd026d', 'request_id': 'req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.500141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01a95b064cf4490bab78eb2fa2fd026d', 'request_id': 'req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.515983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d78eb25fd73444abeaaab742778c7f5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.518441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] The token cache is being invalidate because project 01a95b064cf4490bab78eb2fa2fd026d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:15.518850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.520233 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.522361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1004/2000] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:15 2022] DELETE /identity/v3/projects/01a95b064cf4490bab78eb2fa2fd026d => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:15.523335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:15.524937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:15.527679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d78eb25fd73444abeaaab742778c7f5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.528723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d78eb25fd73444abeaaab742778c7f5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.529636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d78eb25fd73444abeaaab742778c7f5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.538872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f203e847-7a61-44cf-ae4f-1b3ce3187731 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d78eb25fd73444abeaaab742778c7f5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.574823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 997/2001] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:15 2022] DELETE /identity/v3/users/2d78eb25fd73444abeaaab742778c7f5 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:15.576366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a539e16e-ce12-47ed-9583-56ce916abe5c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.576788 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a539e16e-ce12-47ed-9583-56ce916abe5c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.577088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a539e16e-ce12-47ed-9583-56ce916abe5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.605502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a539e16e-ce12-47ed-9583-56ce916abe5c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:15.606731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a539e16e-ce12-47ed-9583-56ce916abe5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:15.625831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:15.627437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e9e23e-646b-4e27-aa5e-8fbfcc10ed09 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:15.726265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 998/2002] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:15.727740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109c906-0363-4ea7-b0f6-309da4238e82 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.728153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109c906-0363-4ea7-b0f6-309da4238e82 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.728537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109c906-0363-4ea7-b0f6-309da4238e82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.731916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:15.732828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.733500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.733837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] PATH_INFO: `/v3/users/28c7737926604698ae2defb96f83434b` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.735138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28c7737926604698ae2defb96f83434b)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:15.738458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:15.752317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5109c906-0363-4ea7-b0f6-309da4238e82 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:15.753775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5109c906-0363-4ea7-b0f6-309da4238e82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:15.763958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 999/2003] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:15.765219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109c906-0363-4ea7-b0f6-309da4238e82 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:15.778202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109c906-0363-4ea7-b0f6-309da4238e82 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:15.783410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49ea239-b86b-439a-ba12-8ed9da9333d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:15.784039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ea239-b86b-439a-ba12-8ed9da9333d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.784320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ea239-b86b-439a-ba12-8ed9da9333d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.784629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ea239-b86b-439a-ba12-8ed9da9333d9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.785306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49ea239-b86b-439a-ba12-8ed9da9333d9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:15.785634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49ea239-b86b-439a-ba12-8ed9da9333d9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:15.787261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49ea239-b86b-439a-ba12-8ed9da9333d9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:15.804267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1000/2004] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:15.804267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee2cb70-9f25-4e36-948d-1fe42dbd5c8c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.804267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee2cb70-9f25-4e36-948d-1fe42dbd5c8c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.804267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee2cb70-9f25-4e36-948d-1fe42dbd5c8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.813165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28c7737926604698ae2defb96f83434b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.815066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:15.824182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28c7737926604698ae2defb96f83434b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.825427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28c7737926604698ae2defb96f83434b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.826219 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28c7737926604698ae2defb96f83434b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.827202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2ee2cb70-9f25-4e36-948d-1fe42dbd5c8c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:15.828923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2ee2cb70-9f25-4e36-948d-1fe42dbd5c8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:15.843237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1001/2005] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:15.843237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee2cb70-9f25-4e36-948d-1fe42dbd5c8c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:15.843541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28c7737926604698ae2defb96f83434b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:15.844019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2ee2cb70-9f25-4e36-948d-1fe42dbd5c8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:15.857175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1005/2006] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:15 2022] DELETE /identity/v3/users/28c7737926604698ae2defb96f83434b => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:15.858331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:15.860866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee2cb70-9f25-4e36-948d-1fe42dbd5c8c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:15.969029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37a147f-70ad-4206-8b25-39eeaddd3b01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:15.969566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37a147f-70ad-4206-8b25-39eeaddd3b01 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.970935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37a147f-70ad-4206-8b25-39eeaddd3b01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.971240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37a147f-70ad-4206-8b25-39eeaddd3b01 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.971962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37a147f-70ad-4206-8b25-39eeaddd3b01 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:15.974256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37a147f-70ad-4206-8b25-39eeaddd3b01 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:15.977494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92e7a57-d8bd-450e-9575-8237f919c9e3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:15.997226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c4d697-c188-4319-81e3-147f6fd2cf3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:15.997807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c4d697-c188-4319-81e3-147f6fd2cf3b admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:15.998156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c4d697-c188-4319-81e3-147f6fd2cf3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:15.998469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c4d697-c188-4319-81e3-147f6fd2cf3b admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:15.999139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c4d697-c188-4319-81e3-147f6fd2cf3b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:15.999501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c4d697-c188-4319-81e3-147f6fd2cf3b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:16.002369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c4d697-c188-4319-81e3-147f6fd2cf3b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:16.031421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1006/2007] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:16.035806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c4d697-c188-4319-81e3-147f6fd2cf3b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:16.039113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1002/2008] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:15 2022] POST /identity/v3/projects => generated 355 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:16.043418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37a147f-70ad-4206-8b25-39eeaddd3b01 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:16.063392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c4d697-c188-4319-81e3-147f6fd2cf3b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:16.233166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37a147f-70ad-4206-8b25-39eeaddd3b01 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:16.243215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa29dd6-57f4-42cf-9c28-5d556a6cbf6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:16.243917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa29dd6-57f4-42cf-9c28-5d556a6cbf6b admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.244248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa29dd6-57f4-42cf-9c28-5d556a6cbf6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.244627 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa29dd6-57f4-42cf-9c28-5d556a6cbf6b admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.245915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa29dd6-57f4-42cf-9c28-5d556a6cbf6b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:16.248222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa29dd6-57f4-42cf-9c28-5d556a6cbf6b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:16.259015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c52dd9-2896-4227-ae6a-93e596cf0f57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:16.261757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c52dd9-2896-4227-ae6a-93e596cf0f57 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.263250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c52dd9-2896-4227-ae6a-93e596cf0f57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.263878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c52dd9-2896-4227-ae6a-93e596cf0f57 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.265184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c52dd9-2896-4227-ae6a-93e596cf0f57 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:16.270401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c52dd9-2896-4227-ae6a-93e596cf0f57 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:16.302443 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1003/2009] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:16 2022] POST /identity/v3/users => generated 278 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:16.305597 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa29dd6-57f4-42cf-9c28-5d556a6cbf6b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:16.310797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2aa29dd6-57f4-42cf-9c28-5d556a6cbf6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:16.333810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:16.345540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa29dd6-57f4-42cf-9c28-5d556a6cbf6b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:16.363315 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1007/2010] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:16 2022] POST /identity/v3/projects => generated 373 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:16.363519 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:16.364224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.364336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c52dd9-2896-4227-ae6a-93e596cf0f57 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:16.364606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.364937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] PATH_INFO: `/v3/projects/474d52be0f6845dbb6043b564d3bc17d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.368553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] RBAC: Authorizing `identity:delete_project(project_id=474d52be0f6845dbb6043b564d3bc17d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:16.373274 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:16.464641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:INSERT=2,keystone:DELETE=3,keystone:SELECT=95 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:16.500899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '474d52be0f6845dbb6043b564d3bc17d', 'request_id': 'req-3614b48d-6364-429f-8ddd-63211a22d39f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:16.512709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c52dd9-2896-4227-ae6a-93e596cf0f57 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:16.514757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '474d52be0f6845dbb6043b564d3bc17d', 'request_id': 'req-3614b48d-6364-429f-8ddd-63211a22d39f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:16.534560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '474d52be0f6845dbb6043b564d3bc17d', 'request_id': 'req-3614b48d-6364-429f-8ddd-63211a22d39f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:16.550909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] The token cache is being invalidate because project 474d52be0f6845dbb6043b564d3bc17d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:16.551858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:16.552759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:16.560982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1004/2011] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:16 2022] DELETE /identity/v3/projects/474d52be0f6845dbb6043b564d3bc17d => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:16.562070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:16.563667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:16.605642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1b9b54-8d6d-485b-aacd-d789a989265a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:16.606713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b9b54-8d6d-485b-aacd-d789a989265a admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.607187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b9b54-8d6d-485b-aacd-d789a989265a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.607827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b9b54-8d6d-485b-aacd-d789a989265a admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.608813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1b9b54-8d6d-485b-aacd-d789a989265a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:16.611533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1b9b54-8d6d-485b-aacd-d789a989265a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:16.630311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1008/2012] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:16.632563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de95b920-5f47-48a0-955a-c2fadb6504b6 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.633320 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de95b920-5f47-48a0-955a-c2fadb6504b6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.633804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de95b920-5f47-48a0-955a-c2fadb6504b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.671033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-de95b920-5f47-48a0-955a-c2fadb6504b6 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:16.673821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-de95b920-5f47-48a0-955a-c2fadb6504b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:16.694580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3614b48d-6364-429f-8ddd-63211a22d39f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:16.701086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69e39a1-c60c-4606-a865-c81633da6934 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:16.701761 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69e39a1-c60c-4606-a865-c81633da6934 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.708841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1009/2013] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:16.709565 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69e39a1-c60c-4606-a865-c81633da6934 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.709565 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69e39a1-c60c-4606-a865-c81633da6934 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.709717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de95b920-5f47-48a0-955a-c2fadb6504b6 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:16.709793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69e39a1-c60c-4606-a865-c81633da6934 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:16.709793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69e39a1-c60c-4606-a865-c81633da6934 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:16.709940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-de95b920-5f47-48a0-955a-c2fadb6504b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:16.728425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de95b920-5f47-48a0-955a-c2fadb6504b6 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:16.734873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f625e9-e8cc-44d5-bdb2-536ff319801d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:16.735814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f625e9-e8cc-44d5-bdb2-536ff319801d admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.736147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f625e9-e8cc-44d5-bdb2-536ff319801d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.736503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f625e9-e8cc-44d5-bdb2-536ff319801d admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.737413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f625e9-e8cc-44d5-bdb2-536ff319801d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:16.739652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f625e9-e8cc-44d5-bdb2-536ff319801d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:16.752126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1010/2014] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:16.753787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9283bfc-8164-457d-b12a-a1aeb1a2b8b2 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.754115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9283bfc-8164-457d-b12a-a1aeb1a2b8b2 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.754391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9283bfc-8164-457d-b12a-a1aeb1a2b8b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.786227 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1005/2015] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:16 2022] POST /identity/v3/users => generated 287 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:16.787839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69e39a1-c60c-4606-a865-c81633da6934 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:16.792140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e9283bfc-8164-457d-b12a-a1aeb1a2b8b2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:16.793432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e9283bfc-8164-457d-b12a-a1aeb1a2b8b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:16.801949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69e39a1-c60c-4606-a865-c81633da6934 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:16.811834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd41955c-b387-4ac5-a87e-cdd2fba96832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:16.812456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd41955c-b387-4ac5-a87e-cdd2fba96832 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.812743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd41955c-b387-4ac5-a87e-cdd2fba96832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.813019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd41955c-b387-4ac5-a87e-cdd2fba96832 admin admin] PATH_INFO: `/v3/projects/40be2936837545b7b56ec0aa07229389/users/97145063952d4576ba68e835424da14c/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.818201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1011/2016] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:16.818201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9283bfc-8164-457d-b12a-a1aeb1a2b8b2 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:16.836535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9283bfc-8164-457d-b12a-a1aeb1a2b8b2 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:16.841378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a610ed2a-2a37-4990-9329-89aab6f3cd06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:16.841999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a610ed2a-2a37-4990-9329-89aab6f3cd06 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.842282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a610ed2a-2a37-4990-9329-89aab6f3cd06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.842561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a610ed2a-2a37-4990-9329-89aab6f3cd06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.843839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a610ed2a-2a37-4990-9329-89aab6f3cd06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:16.845505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a610ed2a-2a37-4990-9329-89aab6f3cd06 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:16.855217 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd41955c-b387-4ac5-a87e-cdd2fba96832 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40be2936837545b7b56ec0aa07229389, user_id=97145063952d4576ba68e835424da14c, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:16.858289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd41955c-b387-4ac5-a87e-cdd2fba96832 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:16.873347 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1012/2017] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:16.876581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a610ed2a-2a37-4990-9329-89aab6f3cd06 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:16.889772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1006/2018] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:16 2022] PUT /identity/v3/projects/40be2936837545b7b56ec0aa07229389/users/97145063952d4576ba68e835424da14c/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:16.896204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a610ed2a-2a37-4990-9329-89aab6f3cd06 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:16.899738 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd41955c-b387-4ac5-a87e-cdd2fba96832 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:16.970465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c9d5e5-32f6-49e4-aeaa-1415842efeb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:16.971179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9d5e5-32f6-49e4-aeaa-1415842efeb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.971436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:16.971877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9d5e5-32f6-49e4-aeaa-1415842efeb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.971877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9d5e5-32f6-49e4-aeaa-1415842efeb1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:16.972483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c9d5e5-32f6-49e4-aeaa-1415842efeb1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:16.972835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c9d5e5-32f6-49e4-aeaa-1415842efeb1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:16.974399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c9d5e5-32f6-49e4-aeaa-1415842efeb1 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:16.997248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1013/2019] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:16.997248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84582786-8864-433e-a981-aa5dbd1345e4 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:16.997248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84582786-8864-433e-a981-aa5dbd1345e4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:16.997594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84582786-8864-433e-a981-aa5dbd1345e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.002119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd41955c-b387-4ac5-a87e-cdd2fba96832 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.011685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4ec0e4-7b64-441a-8524-1b7b5a445926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.012326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4ec0e4-7b64-441a-8524-1b7b5a445926 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.012741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4ec0e4-7b64-441a-8524-1b7b5a445926 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.013016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4ec0e4-7b64-441a-8524-1b7b5a445926 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.013672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4ec0e4-7b64-441a-8524-1b7b5a445926 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.015561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4ec0e4-7b64-441a-8524-1b7b5a445926 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.032702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1007/2020] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:17.032702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4ec0e4-7b64-441a-8524-1b7b5a445926 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.035099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-84582786-8864-433e-a981-aa5dbd1345e4 None None] MFA Rules not processed for user `97145063952d4576ba68e835424da14c`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:17.036490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-84582786-8864-433e-a981-aa5dbd1345e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:17.036895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9e4ec0e4-7b64-441a-8524-1b7b5a445926 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:17.058112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4ec0e4-7b64-441a-8524-1b7b5a445926 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.063597 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.064860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.065162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.065428 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] PATH_INFO: `/v3/projects/c5320b0d28a145bf98d4fab4d73e6edb` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.067455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5320b0d28a145bf98d4fab4d73e6edb)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.069761 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.085536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:17.122724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1014/2021] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:16 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:17.124650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84582786-8864-433e-a981-aa5dbd1345e4 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.137392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5320b0d28a145bf98d4fab4d73e6edb', 'request_id': 'req-e41d1293-206b-4ae0-9c17-e329a50c318b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:17.137997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84582786-8864-433e-a981-aa5dbd1345e4 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.145535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5320b0d28a145bf98d4fab4d73e6edb', 'request_id': 'req-e41d1293-206b-4ae0-9c17-e329a50c318b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:17.156286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5320b0d28a145bf98d4fab4d73e6edb', 'request_id': 'req-e41d1293-206b-4ae0-9c17-e329a50c318b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:17.169038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] The token cache is being invalidate because project c5320b0d28a145bf98d4fab4d73e6edb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:17.169327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:17.169997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:17.172150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1008/2022] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:17 2022] DELETE /identity/v3/projects/c5320b0d28a145bf98d4fab4d73e6edb => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:17.172264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.173599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:17.205593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ec1ac6-50b9-426e-87e2-f7521d3bcfc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.206222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ec1ac6-50b9-426e-87e2-f7521d3bcfc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.206746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ec1ac6-50b9-426e-87e2-f7521d3bcfc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.207057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ec1ac6-50b9-426e-87e2-f7521d3bcfc8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.208167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ec1ac6-50b9-426e-87e2-f7521d3bcfc8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.210135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ec1ac6-50b9-426e-87e2-f7521d3bcfc8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.240986 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41d1293-206b-4ae0-9c17-e329a50c318b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.242468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1015/2023] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:17 2022] POST /identity/v3/projects => generated 381 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:17.243384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ec1ac6-50b9-426e-87e2-f7521d3bcfc8 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.244979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-99ec1ac6-50b9-426e-87e2-f7521d3bcfc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:17.304104 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817111f-c6ec-47fb-9356-ca7632ada0ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.304814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817111f-c6ec-47fb-9356-ca7632ada0ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.305099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817111f-c6ec-47fb-9356-ca7632ada0ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.305378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817111f-c6ec-47fb-9356-ca7632ada0ed admin admin] PATH_INFO: `/v3/projects/23bf7776c99748808781747ac5e59768/users/89b3181dbc014c669356af84d392dc4f/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.312752 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ec1ac6-50b9-426e-87e2-f7521d3bcfc8 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.318558 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551ca94d-333f-400b-b4af-31e3c3817234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.319278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551ca94d-333f-400b-b4af-31e3c3817234 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.319686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551ca94d-333f-400b-b4af-31e3c3817234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.319962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551ca94d-333f-400b-b4af-31e3c3817234 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.320626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551ca94d-333f-400b-b4af-31e3c3817234 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.321098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b817111f-c6ec-47fb-9356-ca7632ada0ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=23bf7776c99748808781747ac5e59768, user_id=89b3181dbc014c669356af84d392dc4f, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.322520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551ca94d-333f-400b-b4af-31e3c3817234 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.324138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b817111f-c6ec-47fb-9356-ca7632ada0ed admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.337699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1009/2024] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:17 2022] PUT /identity/v3/projects/23bf7776c99748808781747ac5e59768/users/89b3181dbc014c669356af84d392dc4f/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:17.339154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817111f-c6ec-47fb-9356-ca7632ada0ed admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.341649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b817111f-c6ec-47fb-9356-ca7632ada0ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:17.346552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:17.358193 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1016/2025] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:17 2022] POST /identity/v3/projects => generated 357 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:17.359381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551ca94d-333f-400b-b4af-31e3c3817234 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.361200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-551ca94d-333f-400b-b4af-31e3c3817234 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:17.409548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817111f-c6ec-47fb-9356-ca7632ada0ed admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.411028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:17.428988 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551ca94d-333f-400b-b4af-31e3c3817234 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.433733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ade6f6-1803-44b3-8b3c-54eef1525e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.434344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ade6f6-1803-44b3-8b3c-54eef1525e0a admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.434621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ade6f6-1803-44b3-8b3c-54eef1525e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.434892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ade6f6-1803-44b3-8b3c-54eef1525e0a admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.435521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ade6f6-1803-44b3-8b3c-54eef1525e0a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.437476 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ade6f6-1803-44b3-8b3c-54eef1525e0a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.460828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1017/2026] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:17 2022] POST /identity/v3/users => generated 291 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:17.462106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f0d6d1-f87a-4c77-875b-83adbcb87a8e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.462447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f0d6d1-f87a-4c77-875b-83adbcb87a8e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.462723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f0d6d1-f87a-4c77-875b-83adbcb87a8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.469956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.470532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.470812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.471083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] PATH_INFO: `/v3/users/b0b78ee0114341738201d8fb6b583677` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.472218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0b78ee0114341738201d8fb6b583677)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.474195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.491496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-18f0d6d1-f87a-4c77-875b-83adbcb87a8e None None] MFA Rules not processed for user `89b3181dbc014c669356af84d392dc4f`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:17.492683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-18f0d6d1-f87a-4c77-875b-83adbcb87a8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:17.523158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0b78ee0114341738201d8fb6b583677'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:17.530829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0b78ee0114341738201d8fb6b583677'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:17.531584 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0b78ee0114341738201d8fb6b583677'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:17.532317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0b78ee0114341738201d8fb6b583677'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:17.539960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0b78ee0114341738201d8fb6b583677'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:17.568232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1010/2027] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:17 2022] DELETE /identity/v3/users/b0b78ee0114341738201d8fb6b583677 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:17.569866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.571472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:17.578862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1018/2028] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:17.581341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f0d6d1-f87a-4c77-875b-83adbcb87a8e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.584042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-18f0d6d1-f87a-4c77-875b-83adbcb87a8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:17.605126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f0d6d1-f87a-4c77-875b-83adbcb87a8e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.614236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:17.660574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf11999-43f4-4035-816c-34fec37472c1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.666605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2477ca-a696-49a1-8eae-2b9def8e8a5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.667217 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2477ca-a696-49a1-8eae-2b9def8e8a5c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.667506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2477ca-a696-49a1-8eae-2b9def8e8a5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.667780 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2477ca-a696-49a1-8eae-2b9def8e8a5c admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.668439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2477ca-a696-49a1-8eae-2b9def8e8a5c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.670402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2477ca-a696-49a1-8eae-2b9def8e8a5c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.679755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192d7ce2-1520-4fcd-afad-f6d07639603a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.681525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192d7ce2-1520-4fcd-afad-f6d07639603a admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.681907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192d7ce2-1520-4fcd-afad-f6d07639603a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.682634 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192d7ce2-1520-4fcd-afad-f6d07639603a admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.683388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192d7ce2-1520-4fcd-afad-f6d07639603a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.685329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192d7ce2-1520-4fcd-afad-f6d07639603a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.694386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1011/2029] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:17 2022] POST /identity/v3/users => generated 285 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:17.695233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2477ca-a696-49a1-8eae-2b9def8e8a5c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.697104 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9f2477ca-a696-49a1-8eae-2b9def8e8a5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:17.697237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1019/2030] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:17.698660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192d7ce2-1520-4fcd-afad-f6d07639603a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.710384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192d7ce2-1520-4fcd-afad-f6d07639603a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.711649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2477ca-a696-49a1-8eae-2b9def8e8a5c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.716010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b1f8ab-1bde-4bf5-ae53-13ee743f940c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.716821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b1f8ab-1bde-4bf5-ae53-13ee743f940c admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.717140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b1f8ab-1bde-4bf5-ae53-13ee743f940c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.717498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ee6d6-0690-409a-aa11-411f283acbf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.717699 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b1f8ab-1bde-4bf5-ae53-13ee743f940c admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.718233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ee6d6-0690-409a-aa11-411f283acbf7 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.718430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b1f8ab-1bde-4bf5-ae53-13ee743f940c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.718599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ee6d6-0690-409a-aa11-411f283acbf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.718982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ee6d6-0690-409a-aa11-411f283acbf7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.719709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ee6d6-0690-409a-aa11-411f283acbf7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.720079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b1f8ab-1bde-4bf5-ae53-13ee743f940c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.721750 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0ee6d6-0690-409a-aa11-411f283acbf7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.734480 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1020/2031] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:17.734733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b1f8ab-1bde-4bf5-ae53-13ee743f940c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.745392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b1f8ab-1bde-4bf5-ae53-13ee743f940c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.750288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262387c2-b347-4c5b-8d6f-47beb03026e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.750890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262387c2-b347-4c5b-8d6f-47beb03026e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.751167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262387c2-b347-4c5b-8d6f-47beb03026e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.751438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262387c2-b347-4c5b-8d6f-47beb03026e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.752224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262387c2-b347-4c5b-8d6f-47beb03026e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.753693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262387c2-b347-4c5b-8d6f-47beb03026e4 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.776086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1012/2032] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:17 2022] POST /identity/v3/projects => generated 369 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:17.776291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1021/2033] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:17.776408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ee6d6-0690-409a-aa11-411f283acbf7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.776506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262387c2-b347-4c5b-8d6f-47beb03026e4 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.787903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262387c2-b347-4c5b-8d6f-47beb03026e4 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.837469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:17.850631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0ee6d6-0690-409a-aa11-411f283acbf7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.855548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b91815-b4bc-431c-9d79-6ef7a5f111a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.856376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b91815-b4bc-431c-9d79-6ef7a5f111a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.856498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b91815-b4bc-431c-9d79-6ef7a5f111a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.856769 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b91815-b4bc-431c-9d79-6ef7a5f111a4 admin admin] PATH_INFO: `/v3/projects/a0d08415dee040e1b607651787c46fe5/users/0f89413df12d4c04aba683955737591e/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.862157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97d352d-61b6-4ab0-9fd8-14c778268c39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.862776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97d352d-61b6-4ab0-9fd8-14c778268c39 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.863054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97d352d-61b6-4ab0-9fd8-14c778268c39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.863319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97d352d-61b6-4ab0-9fd8-14c778268c39 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.863955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97d352d-61b6-4ab0-9fd8-14c778268c39 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.866581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97d352d-61b6-4ab0-9fd8-14c778268c39 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.872279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b91815-b4bc-431c-9d79-6ef7a5f111a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0d08415dee040e1b607651787c46fe5, user_id=0f89413df12d4c04aba683955737591e, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.875032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b91815-b4bc-431c-9d79-6ef7a5f111a4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.891443 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1013/2034] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:17 2022] PUT /identity/v3/projects/a0d08415dee040e1b607651787c46fe5/users/0f89413df12d4c04aba683955737591e/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:17.892064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b91815-b4bc-431c-9d79-6ef7a5f111a4 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.898116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1022/2035] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:17 2022] POST /identity/v3/users => generated 291 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:17.901963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97d352d-61b6-4ab0-9fd8-14c778268c39 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:17.916590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97d352d-61b6-4ab0-9fd8-14c778268c39 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.969003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b91815-b4bc-431c-9d79-6ef7a5f111a4 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:17.973757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e448f8a0-9e1e-454c-a4f7-0fd98999c6b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.974412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e448f8a0-9e1e-454c-a4f7-0fd98999c6b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.974887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e448f8a0-9e1e-454c-a4f7-0fd98999c6b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.975173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e448f8a0-9e1e-454c-a4f7-0fd98999c6b3 admin admin] PATH_INFO: `/v3/projects/27ede370a8cd46089afd0f46e198b4b2/users/2015cc66e43946659b97f73de831fd75/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.982927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8767df09-fafc-4ddf-b48f-8917d028bee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:17.983953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8767df09-fafc-4ddf-b48f-8917d028bee6 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:17.984236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8767df09-fafc-4ddf-b48f-8917d028bee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:17.984549 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8767df09-fafc-4ddf-b48f-8917d028bee6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:17.985203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8767df09-fafc-4ddf-b48f-8917d028bee6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.986812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8767df09-fafc-4ddf-b48f-8917d028bee6 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.989777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e448f8a0-9e1e-454c-a4f7-0fd98999c6b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27ede370a8cd46089afd0f46e198b4b2, user_id=2015cc66e43946659b97f73de831fd75, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:17.992537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e448f8a0-9e1e-454c-a4f7-0fd98999c6b3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:17.997144 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1023/2036] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:17.998155 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8767df09-fafc-4ddf-b48f-8917d028bee6 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.007395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1014/2037] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:17 2022] PUT /identity/v3/projects/27ede370a8cd46089afd0f46e198b4b2/users/2015cc66e43946659b97f73de831fd75/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:18.008459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e448f8a0-9e1e-454c-a4f7-0fd98999c6b3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.011563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8767df09-fafc-4ddf-b48f-8917d028bee6 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.019024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e448f8a0-9e1e-454c-a4f7-0fd98999c6b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.025941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:18.092527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd484f5e-0893-44a5-bd84-b8c037246af3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.093069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd484f5e-0893-44a5-bd84-b8c037246af3 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.093348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd484f5e-0893-44a5-bd84-b8c037246af3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.093616 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd484f5e-0893-44a5-bd84-b8c037246af3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.094270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd484f5e-0893-44a5-bd84-b8c037246af3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.096001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd484f5e-0893-44a5-bd84-b8c037246af3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.099343 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e448f8a0-9e1e-454c-a4f7-0fd98999c6b3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.105248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.106169 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.106523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.106868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] PATH_INFO: `/v3/projects/c61ce3bb85cd454a80fc3ba695875284` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.107278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1024/2038] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:17 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:18.107980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd484f5e-0893-44a5-bd84-b8c037246af3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.108391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c61ce3bb85cd454a80fc3ba695875284)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.110463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.118797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd484f5e-0893-44a5-bd84-b8c037246af3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.124091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da80e0bc-5286-486f-90fe-d635dd74b125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.124858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da80e0bc-5286-486f-90fe-d635dd74b125 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.125059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da80e0bc-5286-486f-90fe-d635dd74b125 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.125339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da80e0bc-5286-486f-90fe-d635dd74b125 admin admin] PATH_INFO: `/v3/projects/a0d08415dee040e1b607651787c46fe5/users/0f89413df12d4c04aba683955737591e/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.132187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da80e0bc-5286-486f-90fe-d635dd74b125 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0d08415dee040e1b607651787c46fe5, user_id=0f89413df12d4c04aba683955737591e, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.136051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da80e0bc-5286-486f-90fe-d635dd74b125 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.162749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1025/2039] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:18 2022] PUT /identity/v3/projects/a0d08415dee040e1b607651787c46fe5/users/0f89413df12d4c04aba683955737591e/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:18.162749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd3666-a24e-4d20-96aa-f31f8fdfbce9 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.162749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd3666-a24e-4d20-96aa-f31f8fdfbce9 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.162749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd3666-a24e-4d20-96aa-f31f8fdfbce9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.170628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c61ce3bb85cd454a80fc3ba695875284', 'request_id': 'req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:18.179357 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c61ce3bb85cd454a80fc3ba695875284', 'request_id': 'req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:18.187562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-64dd3666-a24e-4d20-96aa-f31f8fdfbce9 None None] MFA Rules not processed for user `2015cc66e43946659b97f73de831fd75`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:18.188932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-64dd3666-a24e-4d20-96aa-f31f8fdfbce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.190445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c61ce3bb85cd454a80fc3ba695875284', 'request_id': 'req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:18.210609 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] The token cache is being invalidate because project c61ce3bb85cd454a80fc3ba695875284 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:18.211103 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:18.212219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:18.214539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1015/2040] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:18 2022] DELETE /identity/v3/projects/c61ce3bb85cd454a80fc3ba695875284 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:18.215509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.217012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.241418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:18.276411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1026/2041] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:18 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:18.276411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdca0b9-7e07-4e26-a7e0-791ff9255189 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.276411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdca0b9-7e07-4e26-a7e0-791ff9255189 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.276804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdca0b9-7e07-4e26-a7e0-791ff9255189 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.299838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695ae4fb-6258-49c8-92a7-e2f067a1b6e8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.305562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d2c45d-ca18-4fe8-b4b7-8e44dd3f8ce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.306232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d2c45d-ca18-4fe8-b4b7-8e44dd3f8ce6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.306640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d2c45d-ca18-4fe8-b4b7-8e44dd3f8ce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.306916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d2c45d-ca18-4fe8-b4b7-8e44dd3f8ce6 admin admin] PATH_INFO: `/v3/projects/73647cb651f14be8af8f1d8ce97a0aaf/users/ea85e94e2046421c934c612b05f06093/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.307896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7bdca0b9-7e07-4e26-a7e0-791ff9255189 None None] MFA Rules not processed for user `0f89413df12d4c04aba683955737591e`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:18.309356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7bdca0b9-7e07-4e26-a7e0-791ff9255189 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.323345 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d2c45d-ca18-4fe8-b4b7-8e44dd3f8ce6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73647cb651f14be8af8f1d8ce97a0aaf, user_id=ea85e94e2046421c934c612b05f06093, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.326064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d2c45d-ca18-4fe8-b4b7-8e44dd3f8ce6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.340987 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1016/2042] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:18 2022] PUT /identity/v3/projects/73647cb651f14be8af8f1d8ce97a0aaf/users/ea85e94e2046421c934c612b05f06093/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:18.345956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796e7659-80b9-4e9f-b689-77bbcc00c630 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.346336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796e7659-80b9-4e9f-b689-77bbcc00c630 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.346684 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796e7659-80b9-4e9f-b689-77bbcc00c630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.374668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-796e7659-80b9-4e9f-b689-77bbcc00c630 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:18.376213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-796e7659-80b9-4e9f-b689-77bbcc00c630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.386224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:18.402533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1027/2043] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:18 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:18.403385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdca0b9-7e07-4e26-a7e0-791ff9255189 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.404709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7bdca0b9-7e07-4e26-a7e0-791ff9255189 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.418434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdca0b9-7e07-4e26-a7e0-791ff9255189 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.451603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1017/2044] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:18.451603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796e7659-80b9-4e9f-b689-77bbcc00c630 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.457119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796e7659-80b9-4e9f-b689-77bbcc00c630 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.463262 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d695daa-d486-43c0-bf05-4ec7027af18b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.463995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d695daa-d486-43c0-bf05-4ec7027af18b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.464706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d695daa-d486-43c0-bf05-4ec7027af18b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.465084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d695daa-d486-43c0-bf05-4ec7027af18b admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.465969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d695daa-d486-43c0-bf05-4ec7027af18b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.468091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d695daa-d486-43c0-bf05-4ec7027af18b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.478613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f95cc0-7fba-4fb0-8467-1b3a9a90a207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.479502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1018/2045] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:18.479622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f95cc0-7fba-4fb0-8467-1b3a9a90a207 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.479717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f95cc0-7fba-4fb0-8467-1b3a9a90a207 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.480046 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f95cc0-7fba-4fb0-8467-1b3a9a90a207 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.480783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f95cc0-7fba-4fb0-8467-1b3a9a90a207 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.482248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d695daa-d486-43c0-bf05-4ec7027af18b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.482849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f95cc0-7fba-4fb0-8467-1b3a9a90a207 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.484245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4d695daa-d486-43c0-bf05-4ec7027af18b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.511057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1028/2046] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:18 2022] POST /identity/v3/projects => generated 383 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:18.512459 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98901b16-810a-49f5-b91d-d9466a95e184 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.512819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98901b16-810a-49f5-b91d-d9466a95e184 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.513557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98901b16-810a-49f5-b91d-d9466a95e184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.535331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-98901b16-810a-49f5-b91d-d9466a95e184 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:18.536584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-98901b16-810a-49f5-b91d-d9466a95e184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.557342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d695daa-d486-43c0-bf05-4ec7027af18b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.585010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:18.602112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1029/2047] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:18.602984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98901b16-810a-49f5-b91d-d9466a95e184 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.612034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98901b16-810a-49f5-b91d-d9466a95e184 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.612468 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e2f6ed-7f1d-41dc-8e08-2375a8e87ec6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.613082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e2f6ed-7f1d-41dc-8e08-2375a8e87ec6 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.613455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e2f6ed-7f1d-41dc-8e08-2375a8e87ec6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.613692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e2f6ed-7f1d-41dc-8e08-2375a8e87ec6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.617845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8527884c-2075-4f60-b918-d3e4b3a5c08f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.618493 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8527884c-2075-4f60-b918-d3e4b3a5c08f admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.618767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8527884c-2075-4f60-b918-d3e4b3a5c08f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.619040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8527884c-2075-4f60-b918-d3e4b3a5c08f admin admin] PATH_INFO: `/v3/projects/73647cb651f14be8af8f1d8ce97a0aaf/users/ea85e94e2046421c934c612b05f06093/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.620990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8527884c-2075-4f60-b918-d3e4b3a5c08f admin admin] RBAC: Authorizing `identity:create_grant(project_id=73647cb651f14be8af8f1d8ce97a0aaf, user_id=ea85e94e2046421c934c612b05f06093, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.623689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8527884c-2075-4f60-b918-d3e4b3a5c08f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.636110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1030/2048] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:18 2022] PUT /identity/v3/projects/73647cb651f14be8af8f1d8ce97a0aaf/users/ea85e94e2046421c934c612b05f06093/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:18.637034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8527884c-2075-4f60-b918-d3e4b3a5c08f admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.690000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e2f6ed-7f1d-41dc-8e08-2375a8e87ec6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.691457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e2f6ed-7f1d-41dc-8e08-2375a8e87ec6 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.701606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1019/2049] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:18 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:18.702425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e2f6ed-7f1d-41dc-8e08-2375a8e87ec6 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.712194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e2f6ed-7f1d-41dc-8e08-2375a8e87ec6 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.716874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a879b085-4b93-4eb2-9251-e32148aec666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.717493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a879b085-4b93-4eb2-9251-e32148aec666 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.717777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a879b085-4b93-4eb2-9251-e32148aec666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.718051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a879b085-4b93-4eb2-9251-e32148aec666 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.718693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a879b085-4b93-4eb2-9251-e32148aec666 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:18.718984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8527884c-2075-4f60-b918-d3e4b3a5c08f admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.719097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a879b085-4b93-4eb2-9251-e32148aec666 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.720634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a879b085-4b93-4eb2-9251-e32148aec666 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.725327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42d5b82-29d2-49d3-af35-1a975c076427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.726803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d5b82-29d2-49d3-af35-1a975c076427 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.727166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d5b82-29d2-49d3-af35-1a975c076427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.727472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d5b82-29d2-49d3-af35-1a975c076427 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.728200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42d5b82-29d2-49d3-af35-1a975c076427 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.728554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:18.731113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42d5b82-29d2-49d3-af35-1a975c076427 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.740893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1020/2050] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:18.743204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd70be29-50fb-431a-a081-cdca4c7437f7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.743498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd70be29-50fb-431a-a081-cdca4c7437f7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.744128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd70be29-50fb-431a-a081-cdca4c7437f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.756442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1031/2051] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:18 2022] POST /identity/v3/users => generated 298 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:18.757448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42d5b82-29d2-49d3-af35-1a975c076427 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.772917 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42d5b82-29d2-49d3-af35-1a975c076427 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.776931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bd70be29-50fb-431a-a081-cdca4c7437f7 None None] MFA Rules not processed for user `ea85e94e2046421c934c612b05f06093`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:18.778313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bd70be29-50fb-431a-a081-cdca4c7437f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.780276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88cc598-d5d8-4a07-9b02-55210acb14dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.780985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88cc598-d5d8-4a07-9b02-55210acb14dc admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.781376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88cc598-d5d8-4a07-9b02-55210acb14dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.781622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88cc598-d5d8-4a07-9b02-55210acb14dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.782262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88cc598-d5d8-4a07-9b02-55210acb14dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.784385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88cc598-d5d8-4a07-9b02-55210acb14dc admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.812832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1032/2052] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:18 2022] POST /identity/v3/projects => generated 383 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:18.813795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88cc598-d5d8-4a07-9b02-55210acb14dc admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.854276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1021/2053] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:18 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:18.854934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd70be29-50fb-431a-a081-cdca4c7437f7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.884069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88cc598-d5d8-4a07-9b02-55210acb14dc admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.889307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09ab889-1fff-44d7-9b37-a39403ef795d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.890081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09ab889-1fff-44d7-9b37-a39403ef795d admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.890359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09ab889-1fff-44d7-9b37-a39403ef795d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.890628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09ab889-1fff-44d7-9b37-a39403ef795d admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.891280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09ab889-1fff-44d7-9b37-a39403ef795d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.892954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09ab889-1fff-44d7-9b37-a39403ef795d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.902981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1033/2054] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:18 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:18.903882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09ab889-1fff-44d7-9b37-a39403ef795d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.911322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd70be29-50fb-431a-a081-cdca4c7437f7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.915170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09ab889-1fff-44d7-9b37-a39403ef795d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:18.920076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4eee10-3e9f-4bf6-966f-cc736de9d154 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.920694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4eee10-3e9f-4bf6-966f-cc736de9d154 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.920973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4eee10-3e9f-4bf6-966f-cc736de9d154 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.921248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4eee10-3e9f-4bf6-966f-cc736de9d154 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.921869 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4eee10-3e9f-4bf6-966f-cc736de9d154 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:18.923765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4eee10-3e9f-4bf6-966f-cc736de9d154 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:18.953191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1034/2055] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:18 2022] POST /identity/v3/users => generated 292 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:18.953191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4eee10-3e9f-4bf6-966f-cc736de9d154 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:18.953468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-eb4eee10-3e9f-4bf6-966f-cc736de9d154 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.964084 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b2749d-0046-46a7-b3e9-6eba8d09a445 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:18.964727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b2749d-0046-46a7-b3e9-6eba8d09a445 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:18.965020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b2749d-0046-46a7-b3e9-6eba8d09a445 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:18.965275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b2749d-0046-46a7-b3e9-6eba8d09a445 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:18.967201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-41b2749d-0046-46a7-b3e9-6eba8d09a445 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:18.984519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:19.017985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4eee10-3e9f-4bf6-966f-cc736de9d154 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:19.022526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd03926-821d-4cbe-a87e-fa990e9f2d07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:19.023134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd03926-821d-4cbe-a87e-fa990e9f2d07 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.023412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd03926-821d-4cbe-a87e-fa990e9f2d07 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.023685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd03926-821d-4cbe-a87e-fa990e9f2d07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.025487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6cd03926-821d-4cbe-a87e-fa990e9f2d07 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:19.025714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:19.029599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b2749d-0046-46a7-b3e9-6eba8d09a445 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:19.031496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b2749d-0046-46a7-b3e9-6eba8d09a445 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:19.041747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1022/2056] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:18 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:19.042669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b2749d-0046-46a7-b3e9-6eba8d09a445 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:19.053650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b2749d-0046-46a7-b3e9-6eba8d09a445 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:19.058519 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d79f6-c46a-434c-af91-a80d4998c3d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:19.059178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d79f6-c46a-434c-af91-a80d4998c3d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.059461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d79f6-c46a-434c-af91-a80d4998c3d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.059741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d79f6-c46a-434c-af91-a80d4998c3d6 admin admin] PATH_INFO: `/v3/projects/7a2b99f0284746feb5d01f0bb2227920/users/11c5effebc3a46598565e2f3187d2fba/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.079878 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110d79f6-c46a-434c-af91-a80d4998c3d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a2b99f0284746feb5d01f0bb2227920, user_id=11c5effebc3a46598565e2f3187d2fba, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:19.083153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110d79f6-c46a-434c-af91-a80d4998c3d6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:19.099476 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd03926-821d-4cbe-a87e-fa990e9f2d07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:19.100017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1023/2057] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:19 2022] PUT /identity/v3/projects/7a2b99f0284746feb5d01f0bb2227920/users/11c5effebc3a46598565e2f3187d2fba/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:19.101113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d79f6-c46a-434c-af91-a80d4998c3d6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:19.103998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd03926-821d-4cbe-a87e-fa990e9f2d07 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:19.115896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1035/2058] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:18 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:19.116661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd03926-821d-4cbe-a87e-fa990e9f2d07 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:19.127027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd03926-821d-4cbe-a87e-fa990e9f2d07 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:19.193689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d79f6-c46a-434c-af91-a80d4998c3d6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:19.204892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f99d227-c192-4387-8510-7dcc2c268922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:19.206020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f99d227-c192-4387-8510-7dcc2c268922 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.206327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f99d227-c192-4387-8510-7dcc2c268922 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.207872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f99d227-c192-4387-8510-7dcc2c268922 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.211066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f99d227-c192-4387-8510-7dcc2c268922 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:19.214901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f99d227-c192-4387-8510-7dcc2c268922 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:19.216834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914db9bc-4ddc-4e8b-96b9-6f042e470b53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:19.217352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914db9bc-4ddc-4e8b-96b9-6f042e470b53 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.217628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914db9bc-4ddc-4e8b-96b9-6f042e470b53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.217902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914db9bc-4ddc-4e8b-96b9-6f042e470b53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.218553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914db9bc-4ddc-4e8b-96b9-6f042e470b53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:19.221704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914db9bc-4ddc-4e8b-96b9-6f042e470b53 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:19.234393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1036/2059] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:19 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:19.236526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1024/2060] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:19 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:19.239947 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914db9bc-4ddc-4e8b-96b9-6f042e470b53 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:19.240908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f99d227-c192-4387-8510-7dcc2c268922 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:19.251491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f99d227-c192-4387-8510-7dcc2c268922 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:19.257220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10daff07-2404-4ce2-9255-871f916900d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:19.258224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10daff07-2404-4ce2-9255-871f916900d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.258657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10daff07-2404-4ce2-9255-871f916900d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.259097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10daff07-2404-4ce2-9255-871f916900d7 admin admin] PATH_INFO: `/v3/projects/7a2b99f0284746feb5d01f0bb2227920/users/11c5effebc3a46598565e2f3187d2fba/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.261279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10daff07-2404-4ce2-9255-871f916900d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a2b99f0284746feb5d01f0bb2227920, user_id=11c5effebc3a46598565e2f3187d2fba, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:19.261403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914db9bc-4ddc-4e8b-96b9-6f042e470b53 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:19.264979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10daff07-2404-4ce2-9255-871f916900d7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:19.271221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf1bd4d-859d-4fa6-8a87-dffc0b32d143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:19.273451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf1bd4d-859d-4fa6-8a87-dffc0b32d143 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.273772 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf1bd4d-859d-4fa6-8a87-dffc0b32d143 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.274081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf1bd4d-859d-4fa6-8a87-dffc0b32d143 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.274809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf1bd4d-859d-4fa6-8a87-dffc0b32d143 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:19.277264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf1bd4d-859d-4fa6-8a87-dffc0b32d143 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:19.282118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1025/2061] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:19 2022] PUT /identity/v3/projects/7a2b99f0284746feb5d01f0bb2227920/users/11c5effebc3a46598565e2f3187d2fba/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:19.294023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64217917-f368-48d1-b913-791f7f3146be None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.294367 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64217917-f368-48d1-b913-791f7f3146be None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.294647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64217917-f368-48d1-b913-791f7f3146be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.304531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1037/2062] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:19 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:19.305519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf1bd4d-859d-4fa6-8a87-dffc0b32d143 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:19.316976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf1bd4d-859d-4fa6-8a87-dffc0b32d143 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:19.335506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-64217917-f368-48d1-b913-791f7f3146be None None] MFA Rules not processed for user `11c5effebc3a46598565e2f3187d2fba`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:19.336793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-64217917-f368-48d1-b913-791f7f3146be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:19.426068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5eb0fa-a5b9-4a02-b41b-c3f0fecd7915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:19.427019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5eb0fa-a5b9-4a02-b41b-c3f0fecd7915 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.427627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5eb0fa-a5b9-4a02-b41b-c3f0fecd7915 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.427937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5eb0fa-a5b9-4a02-b41b-c3f0fecd7915 admin admin] PATH_INFO: `/v3/projects/db1abf10c3e24836a161f3035d7e58ca/users/44d8acef214449a085a2173930bb3ab1/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.432062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:19.441537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1026/2063] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:19 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:19.447463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5eb0fa-a5b9-4a02-b41b-c3f0fecd7915 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db1abf10c3e24836a161f3035d7e58ca, user_id=44d8acef214449a085a2173930bb3ab1, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:19.451594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5eb0fa-a5b9-4a02-b41b-c3f0fecd7915 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:19.457759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64217917-f368-48d1-b913-791f7f3146be None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:19.476386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1038/2064] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:19 2022] PUT /identity/v3/projects/db1abf10c3e24836a161f3035d7e58ca/users/44d8acef214449a085a2173930bb3ab1/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:19.479187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5109a3d-600a-498b-a17c-7f6e1e16dbe4 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.479622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5109a3d-600a-498b-a17c-7f6e1e16dbe4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.480039 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5109a3d-600a-498b-a17c-7f6e1e16dbe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.501999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e5109a3d-600a-498b-a17c-7f6e1e16dbe4 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:19.503465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e5109a3d-600a-498b-a17c-7f6e1e16dbe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:19.552553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:19.587914 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64217917-f368-48d1-b913-791f7f3146be None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:19.647390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1039/2065] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:19.654063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5109a3d-600a-498b-a17c-7f6e1e16dbe4 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:19.655846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e5109a3d-600a-498b-a17c-7f6e1e16dbe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:19.674228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504eea98-367b-4f14-9c5d-54b218061fd8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:19.674943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504eea98-367b-4f14-9c5d-54b218061fd8 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.675228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504eea98-367b-4f14-9c5d-54b218061fd8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.675499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504eea98-367b-4f14-9c5d-54b218061fd8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.744244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5109a3d-600a-498b-a17c-7f6e1e16dbe4 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:19.752393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83675acd-8371-4ffe-99b3-b9b6ba11ff5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:19.752393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83675acd-8371-4ffe-99b3-b9b6ba11ff5b service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:19.752393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83675acd-8371-4ffe-99b3-b9b6ba11ff5b service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:19.752393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83675acd-8371-4ffe-99b3-b9b6ba11ff5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:19.763681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504eea98-367b-4f14-9c5d-54b218061fd8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:19.764020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83675acd-8371-4ffe-99b3-b9b6ba11ff5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:19.765441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504eea98-367b-4f14-9c5d-54b218061fd8 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:19.765673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83675acd-8371-4ffe-99b3-b9b6ba11ff5b service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:19.777085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1040/2066] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:19.778118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1027/2067] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:19 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:20.004173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83675acd-8371-4ffe-99b3-b9b6ba11ff5b service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:20.013487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83675acd-8371-4ffe-99b3-b9b6ba11ff5b service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:20.017254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe1991a-2cd7-4039-acad-26581883febd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:20.017957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe1991a-2cd7-4039-acad-26581883febd service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:20.018301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe1991a-2cd7-4039-acad-26581883febd service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:20.018643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe1991a-2cd7-4039-acad-26581883febd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:20.092052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe1991a-2cd7-4039-acad-26581883febd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:20.093541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe1991a-2cd7-4039-acad-26581883febd service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:20.103501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1041/2068] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:20 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:20.431571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504eea98-367b-4f14-9c5d-54b218061fd8 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:20.450800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504eea98-367b-4f14-9c5d-54b218061fd8 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:20.455599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:20.456233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:20.456546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:20.456825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] PATH_INFO: `/v3/users/2015cc66e43946659b97f73de831fd75` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:20.458051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2015cc66e43946659b97f73de831fd75)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:20.460290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:20.515086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2015cc66e43946659b97f73de831fd75'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:20.523349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2015cc66e43946659b97f73de831fd75'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:20.524955 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2015cc66e43946659b97f73de831fd75'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:20.525858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2015cc66e43946659b97f73de831fd75'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:20.544699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2015cc66e43946659b97f73de831fd75'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:20.570788 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1028/2069] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:20 2022] DELETE /identity/v3/users/2015cc66e43946659b97f73de831fd75 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:20.575593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36bbf52-8270-4cb9-a009-3983de13ff3e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:20.575888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36bbf52-8270-4cb9-a009-3983de13ff3e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:20.576249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36bbf52-8270-4cb9-a009-3983de13ff3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:20.596761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e36bbf52-8270-4cb9-a009-3983de13ff3e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:20.598185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e36bbf52-8270-4cb9-a009-3983de13ff3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:20.620147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:20.691762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1042/2070] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:20.706492 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:20.707979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:20.742431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:20.839698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2cb84d-da66-46f7-8031-721f46995cd6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:20.844655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42b098f-fc61-48b8-9319-70f7e37750a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:20.845606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42b098f-fc61-48b8-9319-70f7e37750a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:20.845895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42b098f-fc61-48b8-9319-70f7e37750a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:20.846172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42b098f-fc61-48b8-9319-70f7e37750a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:20.859463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42b098f-fc61-48b8-9319-70f7e37750a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:20.861276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42b098f-fc61-48b8-9319-70f7e37750a0 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:20.888057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1029/2071] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:21.035743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc7311c-cdda-47b1-babd-525bcada2ca8 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:21.037782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc7311c-cdda-47b1-babd-525bcada2ca8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:21.038156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc7311c-cdda-47b1-babd-525bcada2ca8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:21.073358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-dbc7311c-cdda-47b1-babd-525bcada2ca8 None None] MFA Rules not processed for user `44d8acef214449a085a2173930bb3ab1`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:21.074601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-dbc7311c-cdda-47b1-babd-525bcada2ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:21.162319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1043/2072] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:21.170739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42b098f-fc61-48b8-9319-70f7e37750a0 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:21.172661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e42b098f-fc61-48b8-9319-70f7e37750a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:21.290092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42b098f-fc61-48b8-9319-70f7e37750a0 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:21.317978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870784ff-81f1-4174-86bc-939d171fed21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:21.319483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870784ff-81f1-4174-86bc-939d171fed21 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:21.319888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870784ff-81f1-4174-86bc-939d171fed21 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:21.320220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870784ff-81f1-4174-86bc-939d171fed21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:21.332090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870784ff-81f1-4174-86bc-939d171fed21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:21.333843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870784ff-81f1-4174-86bc-939d171fed21 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:21.350198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1030/2073] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:21.538381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc7311c-cdda-47b1-babd-525bcada2ca8 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:21.541690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-dbc7311c-cdda-47b1-babd-525bcada2ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:21.561740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc7311c-cdda-47b1-babd-525bcada2ca8 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:21.568030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:21.569382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:21.569769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:21.570207 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] PATH_INFO: `/v3/projects/27ede370a8cd46089afd0f46e198b4b2` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:21.571524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27ede370a8cd46089afd0f46e198b4b2)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:21.574930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:21.657908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27ede370a8cd46089afd0f46e198b4b2', 'request_id': 'req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:21.666809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27ede370a8cd46089afd0f46e198b4b2', 'request_id': 'req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:21.677469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27ede370a8cd46089afd0f46e198b4b2', 'request_id': 'req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:21.696861 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] The token cache is being invalidate because project 27ede370a8cd46089afd0f46e198b4b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:21.698307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:21.699083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:21.705229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870784ff-81f1-4174-86bc-939d171fed21 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:21.705684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1044/2074] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:21 2022] DELETE /identity/v3/projects/27ede370a8cd46089afd0f46e198b4b2 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:21.723721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870784ff-81f1-4174-86bc-939d171fed21 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:21.869755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:21.870469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:21.870746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:21.871110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] PATH_INFO: `/v3/users/11c5effebc3a46598565e2f3187d2fba` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:21.876660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=11c5effebc3a46598565e2f3187d2fba)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:21.878747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:21.997676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11c5effebc3a46598565e2f3187d2fba'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.009724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:22.011345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:22.028965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11c5effebc3a46598565e2f3187d2fba'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.029576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11c5effebc3a46598565e2f3187d2fba'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.030650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11c5effebc3a46598565e2f3187d2fba'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.058608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7716fbd4-0011-4fa1-96f9-f5f34d3dbe23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11c5effebc3a46598565e2f3187d2fba'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.058783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28042e2-e5d5-41e6-af7a-78f8f76fa2c5 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:22.060034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:22.082556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:22.083562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:22.085815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:22.086358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] PATH_INFO: `/v3/users/89b3181dbc014c669356af84d392dc4f` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:22.088290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=89b3181dbc014c669356af84d392dc4f)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:22.095058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1031/2075] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:21 2022] DELETE /identity/v3/users/11c5effebc3a46598565e2f3187d2fba => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:22.099880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:22.201329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:22.216174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89b3181dbc014c669356af84d392dc4f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.226503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89b3181dbc014c669356af84d392dc4f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.228294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89b3181dbc014c669356af84d392dc4f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.229457 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89b3181dbc014c669356af84d392dc4f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.279484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89b3181dbc014c669356af84d392dc4f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.306412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1045/2076] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:22 2022] DELETE /identity/v3/users/89b3181dbc014c669356af84d392dc4f => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:22.311564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887205ff-e9c0-4dd2-b4b9-a46e160f9f48 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:22.312070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887205ff-e9c0-4dd2-b4b9-a46e160f9f48 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:22.312375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887205ff-e9c0-4dd2-b4b9-a46e160f9f48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:22.346666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-887205ff-e9c0-4dd2-b4b9-a46e160f9f48 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:22.347913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-887205ff-e9c0-4dd2-b4b9-a46e160f9f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:22.447028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1032/2077] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:22.458288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:22.460717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:22.522294 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887205ff-e9c0-4dd2-b4b9-a46e160f9f48 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:22.524190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-887205ff-e9c0-4dd2-b4b9-a46e160f9f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:22.536780 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd6918d-7d5d-4514-acfd-6c2dd202d9ac admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:22.539765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887205ff-e9c0-4dd2-b4b9-a46e160f9f48 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:22.541686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2accc9fe-b911-4297-809b-65052d72fcd9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:22.542477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2accc9fe-b911-4297-809b-65052d72fcd9 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:22.542873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2accc9fe-b911-4297-809b-65052d72fcd9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:22.543245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2accc9fe-b911-4297-809b-65052d72fcd9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:22.545285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:22.545909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:22.546178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:22.546446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] PATH_INFO: `/v3/projects/7a2b99f0284746feb5d01f0bb2227920` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:22.547573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a2b99f0284746feb5d01f0bb2227920)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:22.549525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:22.556272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2accc9fe-b911-4297-809b-65052d72fcd9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:22.558261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2accc9fe-b911-4297-809b-65052d72fcd9 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:22.580418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1046/2078] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:22.607457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a2b99f0284746feb5d01f0bb2227920', 'request_id': 'req-0f69cf7d-c551-4783-a19d-5e038c022b2c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.614545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a2b99f0284746feb5d01f0bb2227920', 'request_id': 'req-0f69cf7d-c551-4783-a19d-5e038c022b2c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.623580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a2b99f0284746feb5d01f0bb2227920', 'request_id': 'req-0f69cf7d-c551-4783-a19d-5e038c022b2c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.640042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] The token cache is being invalidate because project 7a2b99f0284746feb5d01f0bb2227920 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:22.640645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.641395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f69cf7d-c551-4783-a19d-5e038c022b2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:22.643206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1033/2079] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:22 2022] DELETE /identity/v3/projects/7a2b99f0284746feb5d01f0bb2227920 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:22.648876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce2ef53-0080-40be-ae46-812348c96016 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:22.649161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce2ef53-0080-40be-ae46-812348c96016 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:22.649458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce2ef53-0080-40be-ae46-812348c96016 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:22.676291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-4ce2ef53-0080-40be-ae46-812348c96016 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:22.678302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4ce2ef53-0080-40be-ae46-812348c96016 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:22.759587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1047/2080] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:22.762957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122cb83-b744-4ee9-a88f-f3c3b20acb3a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:22.764270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122cb83-b744-4ee9-a88f-f3c3b20acb3a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:22.765255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c122cb83-b744-4ee9-a88f-f3c3b20acb3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:22.790680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c122cb83-b744-4ee9-a88f-f3c3b20acb3a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:22.792607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c122cb83-b744-4ee9-a88f-f3c3b20acb3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:22.802039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1034/2081] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:22.807483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce2ef53-0080-40be-ae46-812348c96016 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:22.818879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce2ef53-0080-40be-ae46-812348c96016 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:22.824444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1470605f-c8a1-4a24-8cd6-f7b86149d073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:22.825055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1470605f-c8a1-4a24-8cd6-f7b86149d073 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:22.825337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1470605f-c8a1-4a24-8cd6-f7b86149d073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:22.825611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1470605f-c8a1-4a24-8cd6-f7b86149d073 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:22.826266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1470605f-c8a1-4a24-8cd6-f7b86149d073 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:22.826567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1470605f-c8a1-4a24-8cd6-f7b86149d073 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:22.828227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1470605f-c8a1-4a24-8cd6-f7b86149d073 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:22.841651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1048/2082] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:22.845817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c122cb83-b744-4ee9-a88f-f3c3b20acb3a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:22.855211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c122cb83-b744-4ee9-a88f-f3c3b20acb3a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:22.861884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fb9e0f-458d-45f6-985d-79421e489c61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:22.862494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fb9e0f-458d-45f6-985d-79421e489c61 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:22.862776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fb9e0f-458d-45f6-985d-79421e489c61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:22.863049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fb9e0f-458d-45f6-985d-79421e489c61 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:22.866630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fb9e0f-458d-45f6-985d-79421e489c61 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:22.866630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fb9e0f-458d-45f6-985d-79421e489c61 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:22.887641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1470605f-c8a1-4a24-8cd6-f7b86149d073 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:22.890023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1470605f-c8a1-4a24-8cd6-f7b86149d073 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:22.908054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1470605f-c8a1-4a24-8cd6-f7b86149d073 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:22.912396 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1035/2083] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:22 2022] POST /identity/v3/projects => generated 377 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:22.916296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fb9e0f-458d-45f6-985d-79421e489c61 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:22.970924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:22.977190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:23.003034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:23.003799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:23.004453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:23.004757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] PATH_INFO: `/v3/projects/23bf7776c99748808781747ac5e59768` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:23.007020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] RBAC: Authorizing `identity:delete_project(project_id=23bf7776c99748808781747ac5e59768)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:23.009076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:23.011610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fb9e0f-458d-45f6-985d-79421e489c61 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:23.016664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7c54a4-4356-4608-8ae8-2f7d3288608c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:23.017388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7c54a4-4356-4608-8ae8-2f7d3288608c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:23.017741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7c54a4-4356-4608-8ae8-2f7d3288608c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:23.018086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7c54a4-4356-4608-8ae8-2f7d3288608c admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:23.018900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7c54a4-4356-4608-8ae8-2f7d3288608c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:23.020939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7c54a4-4356-4608-8ae8-2f7d3288608c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:23.059232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1036/2084] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:22 2022] POST /identity/v3/users => generated 289 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:23.060725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160c8f5d-d064-4a21-84b1-a295b10d383a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:23.061693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160c8f5d-d064-4a21-84b1-a295b10d383a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:23.062175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160c8f5d-d064-4a21-84b1-a295b10d383a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:23.068895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23bf7776c99748808781747ac5e59768', 'request_id': 'req-6c5db196-1134-4443-9036-b0aa2f868c3a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:23.076018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23bf7776c99748808781747ac5e59768', 'request_id': 'req-6c5db196-1134-4443-9036-b0aa2f868c3a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:23.082034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23bf7776c99748808781747ac5e59768', 'request_id': 'req-6c5db196-1134-4443-9036-b0aa2f868c3a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:23.082160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-160c8f5d-d064-4a21-84b1-a295b10d383a None None] MFA Rules not processed for user `44d8acef214449a085a2173930bb3ab1`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:23.083302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-160c8f5d-d064-4a21-84b1-a295b10d383a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:23.094062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] The token cache is being invalidate because project 23bf7776c99748808781747ac5e59768 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:23.094335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:23.094981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:23.096519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1049/2085] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:22 2022] DELETE /identity/v3/projects/23bf7776c99748808781747ac5e59768 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:23.099935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:23.099935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:23.140216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1037/2086] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:23 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:23.144213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160c8f5d-d064-4a21-84b1-a295b10d383a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:23.144213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-160c8f5d-d064-4a21-84b1-a295b10d383a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:23.170733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5db196-1134-4443-9036-b0aa2f868c3a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:23.177185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d652af1-f453-407e-bf1d-c118ec6447dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:23.178920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d652af1-f453-407e-bf1d-c118ec6447dd admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:23.179209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d652af1-f453-407e-bf1d-c118ec6447dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:23.180054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d652af1-f453-407e-bf1d-c118ec6447dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:23.181146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d652af1-f453-407e-bf1d-c118ec6447dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:23.184131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d652af1-f453-407e-bf1d-c118ec6447dd admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:23.201148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1050/2087] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:23 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:23.203240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d652af1-f453-407e-bf1d-c118ec6447dd admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:23.214547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d652af1-f453-407e-bf1d-c118ec6447dd admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:23.218560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160c8f5d-d064-4a21-84b1-a295b10d383a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:23.219464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d52747-2bac-4535-a3cf-eb4bd4921e60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:23.220066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d52747-2bac-4535-a3cf-eb4bd4921e60 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:23.220364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d52747-2bac-4535-a3cf-eb4bd4921e60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:23.220636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d52747-2bac-4535-a3cf-eb4bd4921e60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:23.222615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d52747-2bac-4535-a3cf-eb4bd4921e60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:23.222929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d52747-2bac-4535-a3cf-eb4bd4921e60 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:23.225968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:23.226810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:23.227194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:23.227573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] PATH_INFO: `/v3/users/ea85e94e2046421c934c612b05f06093` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:23.229228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea85e94e2046421c934c612b05f06093)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:23.231381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:23.237897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1051/2088] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:23 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:23.242998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d52747-2bac-4535-a3cf-eb4bd4921e60 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:23.260730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d52747-2bac-4535-a3cf-eb4bd4921e60 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:23.269105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db390b60-c462-4f9d-98f6-a1bfa9e599ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:23.269833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db390b60-c462-4f9d-98f6-a1bfa9e599ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:23.270115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db390b60-c462-4f9d-98f6-a1bfa9e599ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:23.270448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db390b60-c462-4f9d-98f6-a1bfa9e599ea admin admin] PATH_INFO: `/v3/projects/ece01d95a9854799921310006905b9d5/users/fc4a4795eced48058b61553c0fff9fde/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:23.291051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db390b60-c462-4f9d-98f6-a1bfa9e599ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=ece01d95a9854799921310006905b9d5, user_id=fc4a4795eced48058b61553c0fff9fde, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:23.291446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea85e94e2046421c934c612b05f06093'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:23.304387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db390b60-c462-4f9d-98f6-a1bfa9e599ea admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:23.304722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea85e94e2046421c934c612b05f06093'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:23.304722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea85e94e2046421c934c612b05f06093'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:23.304899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea85e94e2046421c934c612b05f06093'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:23.313338 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea85e94e2046421c934c612b05f06093'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:23.322623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=87,keystone:INSERT=4,keystone:DELETE=9 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:23.335812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1052/2089] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:23 2022] PUT /identity/v3/projects/ece01d95a9854799921310006905b9d5/users/fc4a4795eced48058b61553c0fff9fde/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:23.340172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f780e1-f59f-4cdc-a0d5-1f0ec0793b14 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:23.341025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f780e1-f59f-4cdc-a0d5-1f0ec0793b14 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:23.341303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f780e1-f59f-4cdc-a0d5-1f0ec0793b14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:23.354594 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1038/2090] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:23 2022] DELETE /identity/v3/users/ea85e94e2046421c934c612b05f06093 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:23.396821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-54f780e1-f59f-4cdc-a0d5-1f0ec0793b14 None None] MFA Rules not processed for user `fc4a4795eced48058b61553c0fff9fde`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:23.398401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-54f780e1-f59f-4cdc-a0d5-1f0ec0793b14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:23.435677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:23.512399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1053/2091] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:23 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:23.527942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:23.529979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:23.626150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c545710a-b537-4b9a-8673-371c38fdb2f3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:23.640755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86d493f-76d6-4239-a2b1-d981761a19cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:23.641489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d493f-76d6-4239-a2b1-d981761a19cb service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:23.641892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d493f-76d6-4239-a2b1-d981761a19cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:23.642342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d493f-76d6-4239-a2b1-d981761a19cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:23.658607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86d493f-76d6-4239-a2b1-d981761a19cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:23.660395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86d493f-76d6-4239-a2b1-d981761a19cb service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:23.671039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1039/2092] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:23 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:23.753140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f780e1-f59f-4cdc-a0d5-1f0ec0793b14 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:23.754593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-54f780e1-f59f-4cdc-a0d5-1f0ec0793b14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:23.910490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f780e1-f59f-4cdc-a0d5-1f0ec0793b14 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:23.917318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:23.918190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:23.918417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:23.918773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] PATH_INFO: `/v3/projects/73647cb651f14be8af8f1d8ce97a0aaf` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:23.920969 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] RBAC: Authorizing `identity:delete_project(project_id=73647cb651f14be8af8f1d8ce97a0aaf)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:23.924265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:24.090497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73647cb651f14be8af8f1d8ce97a0aaf', 'request_id': 'req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:24.102893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73647cb651f14be8af8f1d8ce97a0aaf', 'request_id': 'req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:24.140981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73647cb651f14be8af8f1d8ce97a0aaf', 'request_id': 'req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:24.161274 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] The token cache is being invalidate because project 73647cb651f14be8af8f1d8ce97a0aaf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:24.161587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:24.163617 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65b0a38d-5ddf-4013-a3b7-28e1b4219caa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:24.168587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1054/2093] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:23 2022] DELETE /identity/v3/projects/73647cb651f14be8af8f1d8ce97a0aaf => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:24.174710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b9178-705b-42d2-afa0-7c1fa002b4f0 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:24.175336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b9178-705b-42d2-afa0-7c1fa002b4f0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:24.175336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b9178-705b-42d2-afa0-7c1fa002b4f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:24.200200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c93b9178-705b-42d2-afa0-7c1fa002b4f0 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:24.201753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c93b9178-705b-42d2-afa0-7c1fa002b4f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:24.269999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1040/2094] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:24.279290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e5b327-8ba6-45ea-af3b-ad148ba3d6b0 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:24.279689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e5b327-8ba6-45ea-af3b-ad148ba3d6b0 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:24.280028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e5b327-8ba6-45ea-af3b-ad148ba3d6b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:24.300964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-74e5b327-8ba6-45ea-af3b-ad148ba3d6b0 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:24.302754 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-74e5b327-8ba6-45ea-af3b-ad148ba3d6b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:24.311739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1055/2095] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:24.315963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b9178-705b-42d2-afa0-7c1fa002b4f0 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:24.326802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b9178-705b-42d2-afa0-7c1fa002b4f0 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:24.331776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47952eac-0459-4819-a120-74252f2c4e08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:24.332509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47952eac-0459-4819-a120-74252f2c4e08 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:24.332858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47952eac-0459-4819-a120-74252f2c4e08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:24.333207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47952eac-0459-4819-a120-74252f2c4e08 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:24.334062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47952eac-0459-4819-a120-74252f2c4e08 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:24.334438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47952eac-0459-4819-a120-74252f2c4e08 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:24.336282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47952eac-0459-4819-a120-74252f2c4e08 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:24.348908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1041/2096] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:24.353557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e5b327-8ba6-45ea-af3b-ad148ba3d6b0 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:24.364483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e5b327-8ba6-45ea-af3b-ad148ba3d6b0 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:24.370869 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec84a07-ac19-42df-8817-5e2ca3611fb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:24.371623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec84a07-ac19-42df-8817-5e2ca3611fb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:24.372009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec84a07-ac19-42df-8817-5e2ca3611fb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:24.372439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec84a07-ac19-42df-8817-5e2ca3611fb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:24.373206 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec84a07-ac19-42df-8817-5e2ca3611fb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:24.375461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec84a07-ac19-42df-8817-5e2ca3611fb1 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:24.406028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1056/2097] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:24 2022] POST /identity/v3/projects => generated 367 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:24.410801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47952eac-0459-4819-a120-74252f2c4e08 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:24.421226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47952eac-0459-4819-a120-74252f2c4e08 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:24.488991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5111708-dcfd-48c4-9d31-2e1f18e1f982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:24.489967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5111708-dcfd-48c4-9d31-2e1f18e1f982 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:24.490298 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5111708-dcfd-48c4-9d31-2e1f18e1f982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:24.490708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5111708-dcfd-48c4-9d31-2e1f18e1f982 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:24.491409 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5111708-dcfd-48c4-9d31-2e1f18e1f982 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:24.493910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5111708-dcfd-48c4-9d31-2e1f18e1f982 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:24.524758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1042/2098] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:24 2022] POST /identity/v3/users => generated 284 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:24.528787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec84a07-ac19-42df-8817-5e2ca3611fb1 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:24.540159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec84a07-ac19-42df-8817-5e2ca3611fb1 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:24.545939 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfeb9961-f548-4815-b67e-3a8c2df4151f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:24.546580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfeb9961-f548-4815-b67e-3a8c2df4151f admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:24.546979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfeb9961-f548-4815-b67e-3a8c2df4151f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:24.547402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfeb9961-f548-4815-b67e-3a8c2df4151f admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:24.548220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfeb9961-f548-4815-b67e-3a8c2df4151f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:24.550129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfeb9961-f548-4815-b67e-3a8c2df4151f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:24.562633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1057/2099] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:24 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:24.567105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5111708-dcfd-48c4-9d31-2e1f18e1f982 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:24.578570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5111708-dcfd-48c4-9d31-2e1f18e1f982 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:24.583375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b67ee8-da20-46c3-9709-36818e1370e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:24.584648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b67ee8-da20-46c3-9709-36818e1370e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:24.584931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b67ee8-da20-46c3-9709-36818e1370e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:24.585204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b67ee8-da20-46c3-9709-36818e1370e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:24.585848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b67ee8-da20-46c3-9709-36818e1370e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:24.588329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b67ee8-da20-46c3-9709-36818e1370e6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:24.590818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:24.605829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1043/2100] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:24 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:24.609666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfeb9961-f548-4815-b67e-3a8c2df4151f admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:24.619916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfeb9961-f548-4815-b67e-3a8c2df4151f admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:24.624555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad99a89b-1f6a-4180-9180-6313cb2d2fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:24.625333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad99a89b-1f6a-4180-9180-6313cb2d2fd8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:24.625623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad99a89b-1f6a-4180-9180-6313cb2d2fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:24.625918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad99a89b-1f6a-4180-9180-6313cb2d2fd8 admin admin] PATH_INFO: `/v3/projects/c67f8f50478749db99129a6b213b4d7f/users/6cc96b5420434be5bedded3b80dfd1fa/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:24.638031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:24.642508 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad99a89b-1f6a-4180-9180-6313cb2d2fd8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c67f8f50478749db99129a6b213b4d7f, user_id=6cc96b5420434be5bedded3b80dfd1fa, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:24.645300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad99a89b-1f6a-4180-9180-6313cb2d2fd8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:24.660739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1058/2101] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:24 2022] PUT /identity/v3/projects/c67f8f50478749db99129a6b213b4d7f/users/6cc96b5420434be5bedded3b80dfd1fa/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:24.671681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdbe3fb-690f-439e-bc97-f88fa7f40e57 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:24.671970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdbe3fb-690f-439e-bc97-f88fa7f40e57 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:24.672245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdbe3fb-690f-439e-bc97-f88fa7f40e57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:24.710246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-efdbe3fb-690f-439e-bc97-f88fa7f40e57 None None] MFA Rules not processed for user `6cc96b5420434be5bedded3b80dfd1fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:24.711496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-efdbe3fb-690f-439e-bc97-f88fa7f40e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:24.791402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1044/2102] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:24 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:24.803773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad99a89b-1f6a-4180-9180-6313cb2d2fd8 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:24.805609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ad99a89b-1f6a-4180-9180-6313cb2d2fd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:24.896294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad99a89b-1f6a-4180-9180-6313cb2d2fd8 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:24.900090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d14a57d-38ef-46aa-93ab-26aa3e9f4b53 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e07b6890d3004dd78faed48a1ab90086', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:24.900730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d14a57d-38ef-46aa-93ab-26aa3e9f4b53 service glance] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:24.901011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d14a57d-38ef-46aa-93ab-26aa3e9f4b53 service glance] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:24.901288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d14a57d-38ef-46aa-93ab-26aa3e9f4b53 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:24.910118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d14a57d-38ef-46aa-93ab-26aa3e9f4b53 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:24.912155 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d14a57d-38ef-46aa-93ab-26aa3e9f4b53 service glance] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:24.921433 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1059/2103] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:24 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:34.788860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=33 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:34.917394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=39,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:41.772858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdbe3fb-690f-439e-bc97-f88fa7f40e57 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:41.774299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-efdbe3fb-690f-439e-bc97-f88fa7f40e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:41.850868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdbe3fb-690f-439e-bc97-f88fa7f40e57 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:41.855652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:41.856284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:41.856617 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:41.856891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] PATH_INFO: `/v3/users/73c365df91254de6a9217dab0280eff3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:41.858028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=73c365df91254de6a9217dab0280eff3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:41.859988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:41.915886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73c365df91254de6a9217dab0280eff3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:41.923722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73c365df91254de6a9217dab0280eff3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:41.924392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73c365df91254de6a9217dab0280eff3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:41.925016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73c365df91254de6a9217dab0280eff3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:41.933112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73c365df91254de6a9217dab0280eff3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:41.957869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1045/2104] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:41 2022] DELETE /identity/v3/users/73c365df91254de6a9217dab0280eff3 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:42.059485 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d14a57d-38ef-46aa-93ab-26aa3e9f4b53 service glance] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:42.061061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9d14a57d-38ef-46aa-93ab-26aa3e9f4b53 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:42.076786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d14a57d-38ef-46aa-93ab-26aa3e9f4b53 service glance] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:42.142380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:42.143118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.143466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.143811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] PATH_INFO: `/v3/users/97145063952d4576ba68e835424da14c` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.145053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] RBAC: Authorizing `identity:delete_user(user_id=97145063952d4576ba68e835424da14c)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:42.146981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:42.171425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:42.172904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:42.201540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec95a588-1020-46b5-a53d-4abfce643af7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:42.204200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '97145063952d4576ba68e835424da14c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:42.207019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:42.208023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.209078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.209470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] PATH_INFO: `/v3/projects/089c1c6e9cb94a19b451d0da50ba9bea` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.211227 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=089c1c6e9cb94a19b451d0da50ba9bea)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:42.213631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:42.213766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '97145063952d4576ba68e835424da14c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:42.214477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '97145063952d4576ba68e835424da14c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:42.215105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '97145063952d4576ba68e835424da14c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:42.229713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-78be1041-aa1a-44c8-bade-38d3e05cbc30 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '97145063952d4576ba68e835424da14c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:42.246753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1060/2105] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:42 2022] DELETE /identity/v3/users/97145063952d4576ba68e835424da14c => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:42.251692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc53533b-6ecc-4389-9e09-ddb02e10c650 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.253474 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc53533b-6ecc-4389-9e09-ddb02e10c650 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.253796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc53533b-6ecc-4389-9e09-ddb02e10c650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.273744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '089c1c6e9cb94a19b451d0da50ba9bea', 'request_id': 'req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:42.281308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '089c1c6e9cb94a19b451d0da50ba9bea', 'request_id': 'req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:42.282514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-fc53533b-6ecc-4389-9e09-ddb02e10c650 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:42.283892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fc53533b-6ecc-4389-9e09-ddb02e10c650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:42.289541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '089c1c6e9cb94a19b451d0da50ba9bea', 'request_id': 'req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:42.301831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] The token cache is being invalidate because project 089c1c6e9cb94a19b451d0da50ba9bea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:42.302132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:42.303545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c19b0526-8fdf-47c8-b531-0e9917ddb6f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:42.305484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1046/2106] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:42 2022] DELETE /identity/v3/projects/089c1c6e9cb94a19b451d0da50ba9bea => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:42.312417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659caeb0-3d3b-493b-a60b-97863d1494b7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.312825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659caeb0-3d3b-493b-a60b-97863d1494b7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.312972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659caeb0-3d3b-493b-a60b-97863d1494b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.321301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:42.343005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-659caeb0-3d3b-493b-a60b-97863d1494b7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:42.344458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-659caeb0-3d3b-493b-a60b-97863d1494b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:42.370439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1061/2107] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:42.383623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc53533b-6ecc-4389-9e09-ddb02e10c650 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:42.385645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fc53533b-6ecc-4389-9e09-ddb02e10c650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:42.423669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:42.438107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1047/2108] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:42.446235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c967c7b-5652-4426-8fdd-dbf8bb7bc9b7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.446540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c967c7b-5652-4426-8fdd-dbf8bb7bc9b7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.447864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c967c7b-5652-4426-8fdd-dbf8bb7bc9b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.468058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1c967c7b-5652-4426-8fdd-dbf8bb7bc9b7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:42.469455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1c967c7b-5652-4426-8fdd-dbf8bb7bc9b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:42.480445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1048/2109] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:42.487592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c967c7b-5652-4426-8fdd-dbf8bb7bc9b7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:42.491575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc53533b-6ecc-4389-9e09-ddb02e10c650 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:42.495287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de8d369-a067-422e-bc7b-2e6f5941bc07 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:42.495992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de8d369-a067-422e-bc7b-2e6f5941bc07 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.496273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de8d369-a067-422e-bc7b-2e6f5941bc07 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.496576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de8d369-a067-422e-bc7b-2e6f5941bc07 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.510788 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c967c7b-5652-4426-8fdd-dbf8bb7bc9b7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:42.515817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a03fbf9-cf11-4610-b7da-2d94f29ccc3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:42.516572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03fbf9-cf11-4610-b7da-2d94f29ccc3f admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.517079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03fbf9-cf11-4610-b7da-2d94f29ccc3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.517379 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de8d369-a067-422e-bc7b-2e6f5941bc07 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:42.518383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03fbf9-cf11-4610-b7da-2d94f29ccc3f admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.518383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a03fbf9-cf11-4610-b7da-2d94f29ccc3f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:42.518757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a03fbf9-cf11-4610-b7da-2d94f29ccc3f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:42.520191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de8d369-a067-422e-bc7b-2e6f5941bc07 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:42.520960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a03fbf9-cf11-4610-b7da-2d94f29ccc3f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:42.539927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1062/2110] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:37:42.542387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1049/2111] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:42.546892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de8d369-a067-422e-bc7b-2e6f5941bc07 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:42.562989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de8d369-a067-422e-bc7b-2e6f5941bc07 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:42.578720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7644f0-e645-49af-a900-d362ac07c420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:42.585608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7644f0-e645-49af-a900-d362ac07c420 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.585776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7644f0-e645-49af-a900-d362ac07c420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.586171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7644f0-e645-49af-a900-d362ac07c420 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.587082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7644f0-e645-49af-a900-d362ac07c420 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:42.589137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7644f0-e645-49af-a900-d362ac07c420 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:42.649738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1063/2112] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:42 2022] POST /identity/v3/projects => generated 375 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:42.657767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a03fbf9-cf11-4610-b7da-2d94f29ccc3f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:42.671333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a03fbf9-cf11-4610-b7da-2d94f29ccc3f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:42.756601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79b0bd5-c6af-4a0e-8fc3-5c856d35c629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:42.757249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79b0bd5-c6af-4a0e-8fc3-5c856d35c629 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.757533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79b0bd5-c6af-4a0e-8fc3-5c856d35c629 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.757812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79b0bd5-c6af-4a0e-8fc3-5c856d35c629 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.758474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79b0bd5-c6af-4a0e-8fc3-5c856d35c629 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:42.761667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79b0bd5-c6af-4a0e-8fc3-5c856d35c629 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:42.796206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1050/2113] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:42 2022] POST /identity/v3/users => generated 288 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:42.797358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7644f0-e645-49af-a900-d362ac07c420 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:42.818908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7644f0-e645-49af-a900-d362ac07c420 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:42.824880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f4b3a-fdb9-47a4-a491-20255f3b3090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:42.826229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f4b3a-fdb9-47a4-a491-20255f3b3090 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.826736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f4b3a-fdb9-47a4-a491-20255f3b3090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.827023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2f4b3a-fdb9-47a4-a491-20255f3b3090 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.827965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2f4b3a-fdb9-47a4-a491-20255f3b3090 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:42.829773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2f4b3a-fdb9-47a4-a491-20255f3b3090 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:42.850058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79b0bd5-c6af-4a0e-8fc3-5c856d35c629 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:42.850920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1064/2114] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:42 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:37:42.863572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79b0bd5-c6af-4a0e-8fc3-5c856d35c629 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:42.872020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2635341a-d5fc-4d49-9270-15dda1f00176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:42.884602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2635341a-d5fc-4d49-9270-15dda1f00176 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.884602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2635341a-d5fc-4d49-9270-15dda1f00176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.884602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2635341a-d5fc-4d49-9270-15dda1f00176 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.884602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2635341a-d5fc-4d49-9270-15dda1f00176 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:42.884602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2635341a-d5fc-4d49-9270-15dda1f00176 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:42.896383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1051/2115] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:42 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:37:42.898078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f4b3a-fdb9-47a4-a491-20255f3b3090 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:42.910065 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2f4b3a-fdb9-47a4-a491-20255f3b3090 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:42.917697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f779d8-0005-4bae-8407-7c6be686d0e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:42.918605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f779d8-0005-4bae-8407-7c6be686d0e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.919162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f779d8-0005-4bae-8407-7c6be686d0e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.919506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f779d8-0005-4bae-8407-7c6be686d0e3 admin admin] PATH_INFO: `/v3/projects/0f9b4bce74cb425993bef9dee7fec2f4/users/b48476466c514a02a01dc280af8b3301/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:42.948883 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f779d8-0005-4bae-8407-7c6be686d0e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f9b4bce74cb425993bef9dee7fec2f4, user_id=b48476466c514a02a01dc280af8b3301, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:42.952254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f779d8-0005-4bae-8407-7c6be686d0e3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:42.975390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2635341a-d5fc-4d49-9270-15dda1f00176 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:42.976600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1065/2116] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:42 2022] PUT /identity/v3/projects/0f9b4bce74cb425993bef9dee7fec2f4/users/b48476466c514a02a01dc280af8b3301/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:42.978238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2635341a-d5fc-4d49-9270-15dda1f00176 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:42.987042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094bd568-a1df-45f4-a54f-6e3f6b2b0914 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:42.987405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094bd568-a1df-45f4-a54f-6e3f6b2b0914 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:42.987762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094bd568-a1df-45f4-a54f-6e3f6b2b0914 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:43.027074 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-094bd568-a1df-45f4-a54f-6e3f6b2b0914 None None] MFA Rules not processed for user `b48476466c514a02a01dc280af8b3301`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:43.029209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-094bd568-a1df-45f4-a54f-6e3f6b2b0914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:43.029752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2635341a-d5fc-4d49-9270-15dda1f00176 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:43.119106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:43.119739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:43.120040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:43.120310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] PATH_INFO: `/v3/projects/40be2936837545b7b56ec0aa07229389` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:43.121478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40be2936837545b7b56ec0aa07229389)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:43.123384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:43.126141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1066/2117] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:42 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:43.126619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:43.149368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094bd568-a1df-45f4-a54f-6e3f6b2b0914 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:43.150955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-094bd568-a1df-45f4-a54f-6e3f6b2b0914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:43.196031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40be2936837545b7b56ec0aa07229389', 'request_id': 'req-5fa06146-847d-4177-8c11-e526cae9d903'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.203127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40be2936837545b7b56ec0aa07229389', 'request_id': 'req-5fa06146-847d-4177-8c11-e526cae9d903'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.219782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40be2936837545b7b56ec0aa07229389', 'request_id': 'req-5fa06146-847d-4177-8c11-e526cae9d903'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.238050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] The token cache is being invalidate because project 40be2936837545b7b56ec0aa07229389 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:43.238353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.239236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.240551 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:43.242048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1052/2118] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:42 2022] DELETE /identity/v3/projects/40be2936837545b7b56ec0aa07229389 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:43.246569 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:43.248003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:43.266739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094bd568-a1df-45f4-a54f-6e3f6b2b0914 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:43.323595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa06146-847d-4177-8c11-e526cae9d903 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:43.328701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:43.329320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:43.329597 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:43.329899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] PATH_INFO: `/v3/users/0f89413df12d4c04aba683955737591e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:43.331031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f89413df12d4c04aba683955737591e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:43.332999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:43.336582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ca8329-99ea-4a25-803a-4fac8c737d4d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:43.337449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ca8329-99ea-4a25-803a-4fac8c737d4d service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:43.337850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ca8329-99ea-4a25-803a-4fac8c737d4d service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:43.338127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ca8329-99ea-4a25-803a-4fac8c737d4d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:43.340303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b1ca8329-99ea-4a25-803a-4fac8c737d4d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:43.388002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f89413df12d4c04aba683955737591e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.398122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f89413df12d4c04aba683955737591e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.398811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f89413df12d4c04aba683955737591e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.399443 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f89413df12d4c04aba683955737591e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.408123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f89413df12d4c04aba683955737591e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.412091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ca8329-99ea-4a25-803a-4fac8c737d4d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:43.413751 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ca8329-99ea-4a25-803a-4fac8c737d4d service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:43.425161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1067/2119] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:43 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:43.429589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1053/2120] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:43 2022] DELETE /identity/v3/users/0f89413df12d4c04aba683955737591e => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:43.672684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ca8329-99ea-4a25-803a-4fac8c737d4d service nova] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:43.674128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b1ca8329-99ea-4a25-803a-4fac8c737d4d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:43.688132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ca8329-99ea-4a25-803a-4fac8c737d4d service nova] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:43.752489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:43.753094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:43.753380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:43.753648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] PATH_INFO: `/v3/users/b48476466c514a02a01dc280af8b3301` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:43.754796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b48476466c514a02a01dc280af8b3301)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:43.757470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:43.765185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:43.767281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:43.784112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503f325-70c8-4df2-a26f-c0c9c8a7cdcb admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:43.789401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:43.790337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:43.790908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:43.791180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] PATH_INFO: `/v3/projects/a0d08415dee040e1b607651787c46fe5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:43.792539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0d08415dee040e1b607651787c46fe5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:43.795876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:43.808720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:43.814774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:43.835580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b48476466c514a02a01dc280af8b3301'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.844560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b48476466c514a02a01dc280af8b3301'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.845278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b48476466c514a02a01dc280af8b3301'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.845962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b48476466c514a02a01dc280af8b3301'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.861584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-29486838-55eb-45ad-b74b-766446f965a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b48476466c514a02a01dc280af8b3301'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.867884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0d08415dee040e1b607651787c46fe5', 'request_id': 'req-06dc9e3a-65cb-4de5-bd51-07ddab413432'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.876991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1068/2121] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:43 2022] DELETE /identity/v3/users/b48476466c514a02a01dc280af8b3301 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:43.881609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f6ee91-e734-419b-ae18-7e25dcb26af7 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:43.881894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f6ee91-e734-419b-ae18-7e25dcb26af7 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:43.882169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f6ee91-e734-419b-ae18-7e25dcb26af7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:43.882272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0d08415dee040e1b607651787c46fe5', 'request_id': 'req-06dc9e3a-65cb-4de5-bd51-07ddab413432'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.889967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0d08415dee040e1b607651787c46fe5', 'request_id': 'req-06dc9e3a-65cb-4de5-bd51-07ddab413432'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.900675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] The token cache is being invalidate because project a0d08415dee040e1b607651787c46fe5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:43.900969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.901609 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-06dc9e3a-65cb-4de5-bd51-07ddab413432 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:43.902661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-29f6ee91-e734-419b-ae18-7e25dcb26af7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:43.903089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1054/2122] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:43 2022] DELETE /identity/v3/projects/a0d08415dee040e1b607651787c46fe5 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:43.906504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-29f6ee91-e734-419b-ae18-7e25dcb26af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:43.907875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848a71c7-bf55-4765-8206-13da514be534 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:43.908156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848a71c7-bf55-4765-8206-13da514be534 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:43.908455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848a71c7-bf55-4765-8206-13da514be534 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:43.929311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-848a71c7-bf55-4765-8206-13da514be534 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:43.930418 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-848a71c7-bf55-4765-8206-13da514be534 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:43.966318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1069/2123] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:43.976185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f6ee91-e734-419b-ae18-7e25dcb26af7 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:43.977651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-29f6ee91-e734-419b-ae18-7e25dcb26af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:43.990669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1055/2124] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:43.999609 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef9f4a6-ef57-4216-9382-707067d5b9cc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:43.999898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef9f4a6-ef57-4216-9382-707067d5b9cc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.000174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef9f4a6-ef57-4216-9382-707067d5b9cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.020099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8ef9f4a6-ef57-4216-9382-707067d5b9cc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:44.021493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8ef9f4a6-ef57-4216-9382-707067d5b9cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:44.029476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1056/2125] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:44.033546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef9f4a6-ef57-4216-9382-707067d5b9cc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.040371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f6ee91-e734-419b-ae18-7e25dcb26af7 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.040371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131a1b28-4b4d-4fd5-8243-ec93f972134c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.040371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131a1b28-4b4d-4fd5-8243-ec93f972134c service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.040371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131a1b28-4b4d-4fd5-8243-ec93f972134c service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.040785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131a1b28-4b4d-4fd5-8243-ec93f972134c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.046376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef9f4a6-ef57-4216-9382-707067d5b9cc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.051509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131a1b28-4b4d-4fd5-8243-ec93f972134c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.051697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e95d6a-9b53-4ee4-aa04-e1229f2f46f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.052852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e95d6a-9b53-4ee4-aa04-e1229f2f46f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.053136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e95d6a-9b53-4ee4-aa04-e1229f2f46f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.053432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e95d6a-9b53-4ee4-aa04-e1229f2f46f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.053711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131a1b28-4b4d-4fd5-8243-ec93f972134c service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.054058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e95d6a-9b53-4ee4-aa04-e1229f2f46f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:44.054362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e95d6a-9b53-4ee4-aa04-e1229f2f46f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.056002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e95d6a-9b53-4ee4-aa04-e1229f2f46f3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.064599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1070/2126] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:37:44.074653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1057/2127] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:44.075740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131a1b28-4b4d-4fd5-8243-ec93f972134c service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.086102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131a1b28-4b4d-4fd5-8243-ec93f972134c service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.092721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2acc4b-ee36-4e8f-986e-ab1d80a3670b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.093393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2acc4b-ee36-4e8f-986e-ab1d80a3670b admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.093739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2acc4b-ee36-4e8f-986e-ab1d80a3670b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.094019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2acc4b-ee36-4e8f-986e-ab1d80a3670b admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.094716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2acc4b-ee36-4e8f-986e-ab1d80a3670b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.096708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2acc4b-ee36-4e8f-986e-ab1d80a3670b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.137065 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1071/2128] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:44 2022] POST /identity/v3/projects => generated 381 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:44.141353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e95d6a-9b53-4ee4-aa04-e1229f2f46f3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.153360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e95d6a-9b53-4ee4-aa04-e1229f2f46f3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.214285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1286bbf8-fe48-42ae-82a4-b29b8eac62ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.215034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1286bbf8-fe48-42ae-82a4-b29b8eac62ea admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.215400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1286bbf8-fe48-42ae-82a4-b29b8eac62ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.215805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1286bbf8-fe48-42ae-82a4-b29b8eac62ea admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.216570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1286bbf8-fe48-42ae-82a4-b29b8eac62ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.218707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1286bbf8-fe48-42ae-82a4-b29b8eac62ea admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.260530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1058/2129] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:44 2022] POST /identity/v3/users => generated 291 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:44.265073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2acc4b-ee36-4e8f-986e-ab1d80a3670b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.275822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2acc4b-ee36-4e8f-986e-ab1d80a3670b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.280697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf5101d-13d1-4a93-98a4-532ae0746d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.281404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf5101d-13d1-4a93-98a4-532ae0746d68 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.281768 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf5101d-13d1-4a93-98a4-532ae0746d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.282089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf5101d-13d1-4a93-98a4-532ae0746d68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.282871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf5101d-13d1-4a93-98a4-532ae0746d68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.284560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf5101d-13d1-4a93-98a4-532ae0746d68 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.295151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1072/2130] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:37:44.299439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1286bbf8-fe48-42ae-82a4-b29b8eac62ea admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.307107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:44.311161 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1286bbf8-fe48-42ae-82a4-b29b8eac62ea admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.317918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3963e8d4-7943-4950-97a1-22aafb361bbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.319153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3963e8d4-7943-4950-97a1-22aafb361bbf admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.319643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3963e8d4-7943-4950-97a1-22aafb361bbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.320043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3963e8d4-7943-4950-97a1-22aafb361bbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.320946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3963e8d4-7943-4950-97a1-22aafb361bbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.323567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3963e8d4-7943-4950-97a1-22aafb361bbf admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.338255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1059/2131] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:44 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:44.339635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf5101d-13d1-4a93-98a4-532ae0746d68 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.350991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf5101d-13d1-4a93-98a4-532ae0746d68 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.356069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75866775-e604-4afe-9536-982f986603b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.356828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75866775-e604-4afe-9536-982f986603b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.357105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75866775-e604-4afe-9536-982f986603b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.357377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75866775-e604-4afe-9536-982f986603b3 admin admin] PATH_INFO: `/v3/projects/60a1e13551fd4f61860742b10e76b54f/users/b1f98d2d70184fd39a53bf6789b44ab5/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.364722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3963e8d4-7943-4950-97a1-22aafb361bbf admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.367077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3963e8d4-7943-4950-97a1-22aafb361bbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:44.376236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75866775-e604-4afe-9536-982f986603b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=60a1e13551fd4f61860742b10e76b54f, user_id=b1f98d2d70184fd39a53bf6789b44ab5, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.380121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75866775-e604-4afe-9536-982f986603b3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.382494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3963e8d4-7943-4950-97a1-22aafb361bbf admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.384397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:44.389088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.389793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.390068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.390376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] PATH_INFO: `/v3/projects/0f9b4bce74cb425993bef9dee7fec2f4` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.391702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f9b4bce74cb425993bef9dee7fec2f4)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.393735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.398779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1073/2132] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:44 2022] PUT /identity/v3/projects/60a1e13551fd4f61860742b10e76b54f/users/b1f98d2d70184fd39a53bf6789b44ab5/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:44.410323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64503087-5c04-408f-8507-d340cb8b330c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.410615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64503087-5c04-408f-8507-d340cb8b330c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.410892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64503087-5c04-408f-8507-d340cb8b330c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.441477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-64503087-5c04-408f-8507-d340cb8b330c None None] MFA Rules not processed for user `b1f98d2d70184fd39a53bf6789b44ab5`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:44.441992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f9b4bce74cb425993bef9dee7fec2f4', 'request_id': 'req-4fa7f158-4e59-4d28-a47e-51d581175c11'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:44.442705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-64503087-5c04-408f-8507-d340cb8b330c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:44.449115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f9b4bce74cb425993bef9dee7fec2f4', 'request_id': 'req-4fa7f158-4e59-4d28-a47e-51d581175c11'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:44.458221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f9b4bce74cb425993bef9dee7fec2f4', 'request_id': 'req-4fa7f158-4e59-4d28-a47e-51d581175c11'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:44.473028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] The token cache is being invalidate because project 0f9b4bce74cb425993bef9dee7fec2f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:44.473326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:44.473989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4fa7f158-4e59-4d28-a47e-51d581175c11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:44.476072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1060/2133] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:44 2022] DELETE /identity/v3/projects/0f9b4bce74cb425993bef9dee7fec2f4 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:44.481038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cff2bae-f774-4176-bb0c-4b9686c52d44 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.481329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cff2bae-f774-4176-bb0c-4b9686c52d44 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.481611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cff2bae-f774-4176-bb0c-4b9686c52d44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.503765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0cff2bae-f774-4176-bb0c-4b9686c52d44 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:44.505020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0cff2bae-f774-4176-bb0c-4b9686c52d44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:44.527988 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1074/2134] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:44 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:44.531405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64503087-5c04-408f-8507-d340cb8b330c None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.533840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-64503087-5c04-408f-8507-d340cb8b330c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:44.573616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1061/2135] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:44.582481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a095c5f-f240-4a6a-931f-7794bab544d5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.582771 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a095c5f-f240-4a6a-931f-7794bab544d5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.583054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a095c5f-f240-4a6a-931f-7794bab544d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.606662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9a095c5f-f240-4a6a-931f-7794bab544d5 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:44.608305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9a095c5f-f240-4a6a-931f-7794bab544d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:44.612902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64503087-5c04-408f-8507-d340cb8b330c None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.618389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bfcefb-246d-4a98-a7d8-76fbfb65c7d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.618560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1062/2136] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:44.619213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bfcefb-246d-4a98-a7d8-76fbfb65c7d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.619506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bfcefb-246d-4a98-a7d8-76fbfb65c7d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.619781 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21bfcefb-246d-4a98-a7d8-76fbfb65c7d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.620455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bfcefb-246d-4a98-a7d8-76fbfb65c7d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.622677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a095c5f-f240-4a6a-931f-7794bab544d5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.622795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21bfcefb-246d-4a98-a7d8-76fbfb65c7d9 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.632206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a095c5f-f240-4a6a-931f-7794bab544d5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.636806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f83488-485b-4665-92fc-02b63f839a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.637395 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f83488-485b-4665-92fc-02b63f839a4a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.637676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f83488-485b-4665-92fc-02b63f839a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.637950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f83488-485b-4665-92fc-02b63f839a4a admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.638622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f83488-485b-4665-92fc-02b63f839a4a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:44.638921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f83488-485b-4665-92fc-02b63f839a4a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.640541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f83488-485b-4665-92fc-02b63f839a4a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.654029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1075/2137] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:44 2022] POST /identity/v3/projects => generated 381 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:44.656507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1063/2138] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:37:44.659213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f83488-485b-4665-92fc-02b63f839a4a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.659908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bfcefb-246d-4a98-a7d8-76fbfb65c7d9 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.673089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f83488-485b-4665-92fc-02b63f839a4a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.744261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:44.745444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21bfcefb-246d-4a98-a7d8-76fbfb65c7d9 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.750890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e4bc43-9f4b-493b-9a55-d2095af6254d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.751250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eacdd1f-75dc-4111-9bff-0ac12f6c5be3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.751494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e4bc43-9f4b-493b-9a55-d2095af6254d admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.751889 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eacdd1f-75dc-4111-9bff-0ac12f6c5be3 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.752030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e4bc43-9f4b-493b-9a55-d2095af6254d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.752188 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eacdd1f-75dc-4111-9bff-0ac12f6c5be3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.752567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eacdd1f-75dc-4111-9bff-0ac12f6c5be3 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.752689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e4bc43-9f4b-493b-9a55-d2095af6254d admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.753196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eacdd1f-75dc-4111-9bff-0ac12f6c5be3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.753367 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e4bc43-9f4b-493b-9a55-d2095af6254d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.755258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eacdd1f-75dc-4111-9bff-0ac12f6c5be3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.755454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e4bc43-9f4b-493b-9a55-d2095af6254d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.775569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:44.785556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1076/2139] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:44 2022] POST /identity/v3/users => generated 297 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:44.788486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eacdd1f-75dc-4111-9bff-0ac12f6c5be3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.795347 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1064/2140] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:44 2022] POST /identity/v3/projects => generated 387 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:44.799487 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e4bc43-9f4b-493b-9a55-d2095af6254d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.800290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eacdd1f-75dc-4111-9bff-0ac12f6c5be3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.862328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aee053a-dc3c-4052-8817-5f5c7a47eb03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.863292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aee053a-dc3c-4052-8817-5f5c7a47eb03 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.863292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aee053a-dc3c-4052-8817-5f5c7a47eb03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.863451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aee053a-dc3c-4052-8817-5f5c7a47eb03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.864092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aee053a-dc3c-4052-8817-5f5c7a47eb03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.865870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aee053a-dc3c-4052-8817-5f5c7a47eb03 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.871156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e4bc43-9f4b-493b-9a55-d2095af6254d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.876424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1038a-a835-49e3-ae5a-3b61195375d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.876620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1077/2141] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:44 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:44.877453 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1038a-a835-49e3-ae5a-3b61195375d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.877742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1038a-a835-49e3-ae5a-3b61195375d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.878019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b1038a-a835-49e3-ae5a-3b61195375d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.878656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b1038a-a835-49e3-ae5a-3b61195375d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.882340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b1038a-a835-49e3-ae5a-3b61195375d6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.882489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aee053a-dc3c-4052-8817-5f5c7a47eb03 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.892954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aee053a-dc3c-4052-8817-5f5c7a47eb03 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.899959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7444834-c9c9-4336-908d-5e60ccc71aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.900694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7444834-c9c9-4336-908d-5e60ccc71aed admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.901242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7444834-c9c9-4336-908d-5e60ccc71aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.901437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7444834-c9c9-4336-908d-5e60ccc71aed admin admin] PATH_INFO: `/v3/projects/7b845937294a4feebed33c831de4c77e/users/94d849cccfe040319acc25d5a7479729/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.911057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1065/2142] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:44 2022] POST /identity/v3/users => generated 294 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:44.912083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1038a-a835-49e3-ae5a-3b61195375d6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.919070 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7444834-c9c9-4336-908d-5e60ccc71aed admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b845937294a4feebed33c831de4c77e, user_id=94d849cccfe040319acc25d5a7479729, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.921872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7444834-c9c9-4336-908d-5e60ccc71aed admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.922679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b1038a-a835-49e3-ae5a-3b61195375d6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:44.927825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e56d7a0-bb41-4e1c-a9bd-a0723c6dff2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:44.928619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e56d7a0-bb41-4e1c-a9bd-a0723c6dff2a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:44.928900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e56d7a0-bb41-4e1c-a9bd-a0723c6dff2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:44.929173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e56d7a0-bb41-4e1c-a9bd-a0723c6dff2a admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:44.929858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e56d7a0-bb41-4e1c-a9bd-a0723c6dff2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:44.931589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e56d7a0-bb41-4e1c-a9bd-a0723c6dff2a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:44.937709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1078/2143] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:44 2022] PUT /identity/v3/projects/7b845937294a4feebed33c831de4c77e/users/94d849cccfe040319acc25d5a7479729/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:44.941716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7444834-c9c9-4336-908d-5e60ccc71aed admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.946622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1066/2144] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:44 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:44.951510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e56d7a0-bb41-4e1c-a9bd-a0723c6dff2a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:44.963904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e56d7a0-bb41-4e1c-a9bd-a0723c6dff2a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.023859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7444834-c9c9-4336-908d-5e60ccc71aed admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.034857 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbaf2bd-d93e-456d-ad1f-dcf907e09031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.035638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbaf2bd-d93e-456d-ad1f-dcf907e09031 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.035929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbaf2bd-d93e-456d-ad1f-dcf907e09031 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.036210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbaf2bd-d93e-456d-ad1f-dcf907e09031 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.037050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbaf2bd-d93e-456d-ad1f-dcf907e09031 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.038839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbaf2bd-d93e-456d-ad1f-dcf907e09031 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.041421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4eead5-e4d1-45e8-a628-d199f31e659e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.041977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4eead5-e4d1-45e8-a628-d199f31e659e admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.042404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4eead5-e4d1-45e8-a628-d199f31e659e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.042693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4eead5-e4d1-45e8-a628-d199f31e659e admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.045443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4eead5-e4d1-45e8-a628-d199f31e659e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.047740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4eead5-e4d1-45e8-a628-d199f31e659e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.056458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1067/2145] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:44 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:45.057513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbaf2bd-d93e-456d-ad1f-dcf907e09031 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.059301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1079/2146] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:44 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:45.063318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4eead5-e4d1-45e8-a628-d199f31e659e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.069715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbaf2bd-d93e-456d-ad1f-dcf907e09031 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.075566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb75815c-c3c6-4038-ac7e-89efc0e0ebd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.075705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4eead5-e4d1-45e8-a628-d199f31e659e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.076234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb75815c-c3c6-4038-ac7e-89efc0e0ebd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.076548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb75815c-c3c6-4038-ac7e-89efc0e0ebd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.076937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb75815c-c3c6-4038-ac7e-89efc0e0ebd0 admin admin] PATH_INFO: `/v3/projects/69e3dd26fc504d75ae393e1194dc2919/users/8ab019bf5a5e4382a519beb8d816dc04/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.080873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe241e99-0fd6-48ad-94b7-eb2ff1284044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.081608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe241e99-0fd6-48ad-94b7-eb2ff1284044 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.081953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe241e99-0fd6-48ad-94b7-eb2ff1284044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.082312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe241e99-0fd6-48ad-94b7-eb2ff1284044 admin admin] PATH_INFO: `/v3/projects/7b845937294a4feebed33c831de4c77e/users/94d849cccfe040319acc25d5a7479729/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.093605 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe241e99-0fd6-48ad-94b7-eb2ff1284044 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b845937294a4feebed33c831de4c77e, user_id=94d849cccfe040319acc25d5a7479729, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.097317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe241e99-0fd6-48ad-94b7-eb2ff1284044 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.100645 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb75815c-c3c6-4038-ac7e-89efc0e0ebd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69e3dd26fc504d75ae393e1194dc2919, user_id=8ab019bf5a5e4382a519beb8d816dc04, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.104630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb75815c-c3c6-4038-ac7e-89efc0e0ebd0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.110765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1080/2147] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:45 2022] PUT /identity/v3/projects/7b845937294a4feebed33c831de4c77e/users/94d849cccfe040319acc25d5a7479729/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:45.123500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1068/2148] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:45 2022] PUT /identity/v3/projects/69e3dd26fc504d75ae393e1194dc2919/users/8ab019bf5a5e4382a519beb8d816dc04/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:45.124928 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31b84e-8a1e-4126-8183-126fb23d6b33 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.125223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31b84e-8a1e-4126-8183-126fb23d6b33 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.125504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db31b84e-8a1e-4126-8183-126fb23d6b33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.134483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e68b5a-c07f-4200-848d-2590cbf9cca1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.134847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e68b5a-c07f-4200-848d-2590cbf9cca1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.135197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e68b5a-c07f-4200-848d-2590cbf9cca1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.155347 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-db31b84e-8a1e-4126-8183-126fb23d6b33 None None] MFA Rules not processed for user `94d849cccfe040319acc25d5a7479729`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:45.156607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-db31b84e-8a1e-4126-8183-126fb23d6b33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:45.176470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-74e68b5a-c07f-4200-848d-2590cbf9cca1 None None] MFA Rules not processed for user `8ab019bf5a5e4382a519beb8d816dc04`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:45.178301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-74e68b5a-c07f-4200-848d-2590cbf9cca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:45.202156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:45.217266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:45.259178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1081/2149] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:45 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:45.264728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db31b84e-8a1e-4126-8183-126fb23d6b33 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.265080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1069/2150] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:45 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:45.266164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-db31b84e-8a1e-4126-8183-126fb23d6b33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:45.270352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e68b5a-c07f-4200-848d-2590cbf9cca1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.351937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db31b84e-8a1e-4126-8183-126fb23d6b33 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.355700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e68b5a-c07f-4200-848d-2590cbf9cca1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.356009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab202991-0314-4ed3-9620-70f135ba0915 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.356719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab202991-0314-4ed3-9620-70f135ba0915 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.357003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab202991-0314-4ed3-9620-70f135ba0915 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.357275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab202991-0314-4ed3-9620-70f135ba0915 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.361617 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327cfae7-6fb6-40fa-8dcb-cf216f59b981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.362320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327cfae7-6fb6-40fa-8dcb-cf216f59b981 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.362720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327cfae7-6fb6-40fa-8dcb-cf216f59b981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.363207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327cfae7-6fb6-40fa-8dcb-cf216f59b981 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.363750 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327cfae7-6fb6-40fa-8dcb-cf216f59b981 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.365813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327cfae7-6fb6-40fa-8dcb-cf216f59b981 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.368381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab202991-0314-4ed3-9620-70f135ba0915 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.369992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab202991-0314-4ed3-9620-70f135ba0915 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.384199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1082/2151] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:45 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:45.403963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1070/2152] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:45 2022] POST /identity/v3/projects => generated 385 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:45.408334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab202991-0314-4ed3-9620-70f135ba0915 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.420990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab202991-0314-4ed3-9620-70f135ba0915 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.492530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0c2860-6afa-4cdf-a0ee-cc97998a678e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.493169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0c2860-6afa-4cdf-a0ee-cc97998a678e admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.493432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0c2860-6afa-4cdf-a0ee-cc97998a678e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.493708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0c2860-6afa-4cdf-a0ee-cc97998a678e admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.494357 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0c2860-6afa-4cdf-a0ee-cc97998a678e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.496533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0c2860-6afa-4cdf-a0ee-cc97998a678e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.521973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1083/2153] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:45 2022] POST /identity/v3/users => generated 299 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:45.523095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327cfae7-6fb6-40fa-8dcb-cf216f59b981 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.533785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327cfae7-6fb6-40fa-8dcb-cf216f59b981 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.539265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70203b27-48a2-4f19-8ffb-633bb0950a5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.539867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70203b27-48a2-4f19-8ffb-633bb0950a5b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.540152 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70203b27-48a2-4f19-8ffb-633bb0950a5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.540517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70203b27-48a2-4f19-8ffb-633bb0950a5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.541163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70203b27-48a2-4f19-8ffb-633bb0950a5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.542800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70203b27-48a2-4f19-8ffb-633bb0950a5b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.554579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1071/2154] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:45.559115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0c2860-6afa-4cdf-a0ee-cc97998a678e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.571325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0c2860-6afa-4cdf-a0ee-cc97998a678e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.577585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d613f06e-9033-4bb4-b34c-52ea8f4d6a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.578257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d613f06e-9033-4bb4-b34c-52ea8f4d6a45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.578538 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d613f06e-9033-4bb4-b34c-52ea8f4d6a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.578813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d613f06e-9033-4bb4-b34c-52ea8f4d6a45 admin admin] PATH_INFO: `/v3/projects/972e35f87282478ca429b99d33b1e201/users/12f5d0a249b3459c83b249c93069943a/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.599852 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d613f06e-9033-4bb4-b34c-52ea8f4d6a45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=972e35f87282478ca429b99d33b1e201, user_id=12f5d0a249b3459c83b249c93069943a, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.602905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d613f06e-9033-4bb4-b34c-52ea8f4d6a45 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.618819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1084/2155] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:45 2022] PUT /identity/v3/projects/972e35f87282478ca429b99d33b1e201/users/12f5d0a249b3459c83b249c93069943a/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:45.623238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70203b27-48a2-4f19-8ffb-633bb0950a5b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.634577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70203b27-48a2-4f19-8ffb-633bb0950a5b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.701393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d679b3ce-d092-4759-9765-5c698e953e3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.702037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d679b3ce-d092-4759-9765-5c698e953e3e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.702344 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d679b3ce-d092-4759-9765-5c698e953e3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.702619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d679b3ce-d092-4759-9765-5c698e953e3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.703274 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d679b3ce-d092-4759-9765-5c698e953e3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.704925 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d679b3ce-d092-4759-9765-5c698e953e3e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.715484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1072/2156] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:45 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:45.719769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d613f06e-9033-4bb4-b34c-52ea8f4d6a45 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.732750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d613f06e-9033-4bb4-b34c-52ea8f4d6a45 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.740560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e84db9-1294-478e-a6c2-b3c8582cfe07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.741230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e84db9-1294-478e-a6c2-b3c8582cfe07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.741508 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e84db9-1294-478e-a6c2-b3c8582cfe07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.741787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e84db9-1294-478e-a6c2-b3c8582cfe07 admin admin] PATH_INFO: `/v3/projects/972e35f87282478ca429b99d33b1e201/users/12f5d0a249b3459c83b249c93069943a/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.743753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e84db9-1294-478e-a6c2-b3c8582cfe07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=972e35f87282478ca429b99d33b1e201, user_id=12f5d0a249b3459c83b249c93069943a, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.746540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e84db9-1294-478e-a6c2-b3c8582cfe07 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.759499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d679b3ce-d092-4759-9765-5c698e953e3e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.773826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1085/2157] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:45 2022] PUT /identity/v3/projects/972e35f87282478ca429b99d33b1e201/users/12f5d0a249b3459c83b249c93069943a/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:45.782191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667b2963-e668-4d94-b6e8-04b5a5f20401 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.782489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667b2963-e668-4d94-b6e8-04b5a5f20401 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.782743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667b2963-e668-4d94-b6e8-04b5a5f20401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.809800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:45.825773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-667b2963-e668-4d94-b6e8-04b5a5f20401 None None] MFA Rules not processed for user `12f5d0a249b3459c83b249c93069943a`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:45.826993 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-667b2963-e668-4d94-b6e8-04b5a5f20401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:45.841832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d679b3ce-d092-4759-9765-5c698e953e3e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.853431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:45.904883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166c5a8-175a-4af2-bfbb-b73aad0d395f tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94d849cccfe040319acc25d5a7479729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7b845937294a4feebed33c831de4c77e', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.905682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166c5a8-175a-4af2-bfbb-b73aad0d395f tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.905968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166c5a8-175a-4af2-bfbb-b73aad0d395f tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.906244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166c5a8-175a-4af2-bfbb-b73aad0d395f tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.906887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166c5a8-175a-4af2-bfbb-b73aad0d395f tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:45.906997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1086/2158] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:45 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:45.907193 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166c5a8-175a-4af2-bfbb-b73aad0d395f tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.908822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166c5a8-175a-4af2-bfbb-b73aad0d395f tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.920479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667b2963-e668-4d94-b6e8-04b5a5f20401 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.926679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1073/2159] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:37:45.931122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166c5a8-175a-4af2-bfbb-b73aad0d395f tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:45.940912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166c5a8-175a-4af2-bfbb-b73aad0d395f tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.946471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd2a5b3-5d39-4529-937e-2231134f3115 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94d849cccfe040319acc25d5a7479729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7b845937294a4feebed33c831de4c77e', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:45.947055 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd2a5b3-5d39-4529-937e-2231134f3115 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:45.947631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd2a5b3-5d39-4529-937e-2231134f3115 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:45.947895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd2a5b3-5d39-4529-937e-2231134f3115 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:45.948556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd2a5b3-5d39-4529-937e-2231134f3115 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:45.950544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd2a5b3-5d39-4529-937e-2231134f3115 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:45.984112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667b2963-e668-4d94-b6e8-04b5a5f20401 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:45.989412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1074/2160] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:45 2022] POST /identity/v3/projects => generated 355 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:45.991178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd2a5b3-5d39-4529-937e-2231134f3115 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:46.038918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72472d7-2a27-413d-bcdd-4814bb83de38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:46.039130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72472d7-2a27-413d-bcdd-4814bb83de38 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:46.039296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72472d7-2a27-413d-bcdd-4814bb83de38 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:46.039568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72472d7-2a27-413d-bcdd-4814bb83de38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:46.061264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd2a5b3-5d39-4529-937e-2231134f3115 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:46.066606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df74a09-08a4-4572-a309-535d109da0ce tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94d849cccfe040319acc25d5a7479729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7b845937294a4feebed33c831de4c77e', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:46.067431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df74a09-08a4-4572-a309-535d109da0ce tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:46.067864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df74a09-08a4-4572-a309-535d109da0ce tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:46.068779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df74a09-08a4-4572-a309-535d109da0ce tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:46.069414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df74a09-08a4-4572-a309-535d109da0ce tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:46.069725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df74a09-08a4-4572-a309-535d109da0ce tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:46.071456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df74a09-08a4-4572-a309-535d109da0ce tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:46.086075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1075/2161] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:46.106826 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72472d7-2a27-413d-bcdd-4814bb83de38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:46.108272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72472d7-2a27-413d-bcdd-4814bb83de38 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:46.122783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df74a09-08a4-4572-a309-535d109da0ce tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:46.125681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1087/2162] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:45 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:37:46.140184 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df74a09-08a4-4572-a309-535d109da0ce tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:46.145644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '94d849cccfe040319acc25d5a7479729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7b845937294a4feebed33c831de4c77e', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:46.146056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:46.146347 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:46.146660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] PATH_INFO: `/v3/projects/98f585143f5f40aa967597ae87afbd29` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:46.147968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: Authorizing `identity:delete_project(project_id=98f585143f5f40aa967597ae87afbd29)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:46.150151 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:46.166081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72472d7-2a27-413d-bcdd-4814bb83de38 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:46.201105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:46.210138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:46.220115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98f585143f5f40aa967597ae87afbd29', 'request_id': 'req-0298e459-921a-40c2-b112-4819670f9669'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.229213 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98f585143f5f40aa967597ae87afbd29', 'request_id': 'req-0298e459-921a-40c2-b112-4819670f9669'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.238546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98f585143f5f40aa967597ae87afbd29', 'request_id': 'req-0298e459-921a-40c2-b112-4819670f9669'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.245526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72472d7-2a27-413d-bcdd-4814bb83de38 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:46.256849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] The token cache is being invalidate because project 98f585143f5f40aa967597ae87afbd29 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:46.258040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.258937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0298e459-921a-40c2-b112-4819670f9669 tempest-VolumeQuotaClassesTest-1050502356 tempest-VolumeQuotaClassesTest-1050502356-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.261238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1076/2163] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:46 2022] DELETE /identity/v3/projects/98f585143f5f40aa967597ae87afbd29 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:46.330961 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:46.331624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:46.331985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:46.332176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] PATH_INFO: `/v3/users/8ab019bf5a5e4382a519beb8d816dc04` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:46.333390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ab019bf5a5e4382a519beb8d816dc04)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:46.335801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:46.389636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ab019bf5a5e4382a519beb8d816dc04'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.398536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ab019bf5a5e4382a519beb8d816dc04'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.399326 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ab019bf5a5e4382a519beb8d816dc04'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.402169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ab019bf5a5e4382a519beb8d816dc04'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.417731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ab019bf5a5e4382a519beb8d816dc04'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.434249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1088/2164] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:46 2022] DELETE /identity/v3/users/8ab019bf5a5e4382a519beb8d816dc04 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:46.437776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed75efc-47cc-4442-8943-e0f20b27e7aa None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:46.438201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed75efc-47cc-4442-8943-e0f20b27e7aa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:46.438539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed75efc-47cc-4442-8943-e0f20b27e7aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:46.468854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5ed75efc-47cc-4442-8943-e0f20b27e7aa None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:46.470281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5ed75efc-47cc-4442-8943-e0f20b27e7aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:46.522672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:46.524142 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:46.549785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1077/2165] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:46.562502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed75efc-47cc-4442-8943-e0f20b27e7aa None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:46.566087 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5ed75efc-47cc-4442-8943-e0f20b27e7aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:46.596968 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:46.614079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256cd897-0aa6-444a-939d-227d9bc9005a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:46.620647 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:46.621285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:46.621564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:46.621836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] PATH_INFO: `/v3/users/b1f98d2d70184fd39a53bf6789b44ab5` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:46.623052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1f98d2d70184fd39a53bf6789b44ab5)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:46.625508 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:46.643028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed75efc-47cc-4442-8943-e0f20b27e7aa None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:46.648217 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31f0a2f-b66e-4e65-b64c-c5d565bc8e02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:46.648889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f0a2f-b66e-4e65-b64c-c5d565bc8e02 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:46.649170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f0a2f-b66e-4e65-b64c-c5d565bc8e02 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:46.649443 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f0a2f-b66e-4e65-b64c-c5d565bc8e02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:46.658883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31f0a2f-b66e-4e65-b64c-c5d565bc8e02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:46.660491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31f0a2f-b66e-4e65-b64c-c5d565bc8e02 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:46.673772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1078/2166] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:46.674773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31f0a2f-b66e-4e65-b64c-c5d565bc8e02 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:46.676971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a31f0a2f-b66e-4e65-b64c-c5d565bc8e02 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:46.677651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1f98d2d70184fd39a53bf6789b44ab5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.685804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1f98d2d70184fd39a53bf6789b44ab5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.686469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1f98d2d70184fd39a53bf6789b44ab5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.687155 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1f98d2d70184fd39a53bf6789b44ab5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.692128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31f0a2f-b66e-4e65-b64c-c5d565bc8e02 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:46.699156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:46.699846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:46.700129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:46.700440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] PATH_INFO: `/v3/users/fc4a4795eced48058b61553c0fff9fde` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:46.701627 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc4a4795eced48058b61553c0fff9fde)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:46.703571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:46.706801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-3657fc34-ba98-402b-a712-01fbf155436b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1f98d2d70184fd39a53bf6789b44ab5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.723122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1089/2167] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:46 2022] DELETE /identity/v3/users/b1f98d2d70184fd39a53bf6789b44ab5 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:46.727776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f777d0-f403-4dc0-ba94-556b0e0b18be None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:46.729171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f777d0-f403-4dc0-ba94-556b0e0b18be None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:46.729533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f777d0-f403-4dc0-ba94-556b0e0b18be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:46.750254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-63f777d0-f403-4dc0-ba94-556b0e0b18be None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:46.751507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-63f777d0-f403-4dc0-ba94-556b0e0b18be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:46.777432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc4a4795eced48058b61553c0fff9fde'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.786999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc4a4795eced48058b61553c0fff9fde'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.788126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc4a4795eced48058b61553c0fff9fde'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.788823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc4a4795eced48058b61553c0fff9fde'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.797389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-bafc837d-6e89-4a61-adf1-96ecbd312b6b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc4a4795eced48058b61553c0fff9fde'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:46.804716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:46.824101 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1079/2168] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:46 2022] DELETE /identity/v3/users/fc4a4795eced48058b61553c0fff9fde => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:46.827949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643f0312-3ab4-4bc0-bf8c-1f0de625866c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:46.828863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643f0312-3ab4-4bc0-bf8c-1f0de625866c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:46.829141 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643f0312-3ab4-4bc0-bf8c-1f0de625866c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:46.846310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1090/2169] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:46.853825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f777d0-f403-4dc0-ba94-556b0e0b18be None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:46.856970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-63f777d0-f403-4dc0-ba94-556b0e0b18be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:46.866144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-643f0312-3ab4-4bc0-bf8c-1f0de625866c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:46.869372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-643f0312-3ab4-4bc0-bf8c-1f0de625866c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:46.944226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f777d0-f403-4dc0-ba94-556b0e0b18be None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:46.955040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1080/2170] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:46.967136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643f0312-3ab4-4bc0-bf8c-1f0de625866c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:46.968822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-643f0312-3ab4-4bc0-bf8c-1f0de625866c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:46.980148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:47.014555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ec0034-3d97-4946-90d9-08e760cf00e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:47.015138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ec0034-3d97-4946-90d9-08e760cf00e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:47.015426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ec0034-3d97-4946-90d9-08e760cf00e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:47.015701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ec0034-3d97-4946-90d9-08e760cf00e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:47.030027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ec0034-3d97-4946-90d9-08e760cf00e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:47.031564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ec0034-3d97-4946-90d9-08e760cf00e9 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:47.040404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643f0312-3ab4-4bc0-bf8c-1f0de625866c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:47.043952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1091/2171] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:47.045224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ec0034-3d97-4946-90d9-08e760cf00e9 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:47.046280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b859e063-4e0b-402b-8074-6e3181b0e724 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:47.046992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b859e063-4e0b-402b-8074-6e3181b0e724 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:47.047339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b859e063-4e0b-402b-8074-6e3181b0e724 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:47.047842 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b859e063-4e0b-402b-8074-6e3181b0e724 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:47.054404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-25ec0034-3d97-4946-90d9-08e760cf00e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:47.067655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b859e063-4e0b-402b-8074-6e3181b0e724 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:47.070028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b859e063-4e0b-402b-8074-6e3181b0e724 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:47.071127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ec0034-3d97-4946-90d9-08e760cf00e9 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:47.077191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:47.078089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:47.078424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:47.078823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] PATH_INFO: `/v3/projects/69e3dd26fc504d75ae393e1194dc2919` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:47.080430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=69e3dd26fc504d75ae393e1194dc2919)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:47.082445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:47.082867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1081/2172] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:37:47.153489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69e3dd26fc504d75ae393e1194dc2919', 'request_id': 'req-60bc7736-e6c4-4e90-b77c-e31e79a809ef'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.163731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69e3dd26fc504d75ae393e1194dc2919', 'request_id': 'req-60bc7736-e6c4-4e90-b77c-e31e79a809ef'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.173221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69e3dd26fc504d75ae393e1194dc2919', 'request_id': 'req-60bc7736-e6c4-4e90-b77c-e31e79a809ef'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.190809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] The token cache is being invalidate because project 69e3dd26fc504d75ae393e1194dc2919 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:47.191110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.191969 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.194256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1092/2173] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:47 2022] DELETE /identity/v3/projects/69e3dd26fc504d75ae393e1194dc2919 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:47.199550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b859e063-4e0b-402b-8074-6e3181b0e724 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:47.211709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b859e063-4e0b-402b-8074-6e3181b0e724 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:47.311637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:47.312288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:47.313082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:47.313366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] PATH_INFO: `/v3/users/12f5d0a249b3459c83b249c93069943a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:47.314804 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=12f5d0a249b3459c83b249c93069943a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:47.316846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:47.369517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=7,keystone:SELECT=93 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:47.398739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12f5d0a249b3459c83b249c93069943a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.403300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:47.407868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:47.419329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12f5d0a249b3459c83b249c93069943a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.420138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12f5d0a249b3459c83b249c93069943a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.420809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12f5d0a249b3459c83b249c93069943a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.440973 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12f5d0a249b3459c83b249c93069943a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.451815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bc7736-e6c4-4e90-b77c-e31e79a809ef admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:47.474465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:47.476115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:47.476716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:47.477168 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] PATH_INFO: `/v3/projects/ece01d95a9854799921310006905b9d5` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:47.480658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1082/2174] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:47 2022] DELETE /identity/v3/users/12f5d0a249b3459c83b249c93069943a => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:47.481622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:47.483641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:47.494238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ece01d95a9854799921310006905b9d5)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:47.498325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:47.572122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ece01d95a9854799921310006905b9d5', 'request_id': 'req-a54226dd-4de7-460c-a4f6-f50d005797b9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.580377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ece01d95a9854799921310006905b9d5', 'request_id': 'req-a54226dd-4de7-460c-a4f6-f50d005797b9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.588947 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ece01d95a9854799921310006905b9d5', 'request_id': 'req-a54226dd-4de7-460c-a4f6-f50d005797b9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.589416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf285d6a-0467-43c1-8744-3bab23fe2c3a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:47.612124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] The token cache is being invalidate because project ece01d95a9854799921310006905b9d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:47.612687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.613487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a54226dd-4de7-460c-a4f6-f50d005797b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.619978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1093/2175] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:47 2022] DELETE /identity/v3/projects/ece01d95a9854799921310006905b9d5 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:47.622458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b606664-6c7a-4146-a521-00f97b76660e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:47.622905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b606664-6c7a-4146-a521-00f97b76660e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:47.623330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b606664-6c7a-4146-a521-00f97b76660e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:47.647439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-3b606664-6c7a-4146-a521-00f97b76660e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:47.649159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3b606664-6c7a-4146-a521-00f97b76660e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:47.667756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:47.668570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:47.672542 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:47.672925 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] PATH_INFO: `/v3/projects/60a1e13551fd4f61860742b10e76b54f` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:47.674312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=60a1e13551fd4f61860742b10e76b54f)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:47.676267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:47.692030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:47.734153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1094/2176] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:47.744716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cf8974-3905-4bc6-ae13-12be086addc7 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:47.745132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cf8974-3905-4bc6-ae13-12be086addc7 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:47.746323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cf8974-3905-4bc6-ae13-12be086addc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:47.754047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '60a1e13551fd4f61860742b10e76b54f', 'request_id': 'req-456d0590-dec6-40e2-8910-7aab004129bc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.761901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '60a1e13551fd4f61860742b10e76b54f', 'request_id': 'req-456d0590-dec6-40e2-8910-7aab004129bc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.765406 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:47.776994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-07cf8974-3905-4bc6-ae13-12be086addc7 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:47.778509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-07cf8974-3905-4bc6-ae13-12be086addc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:47.782339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '60a1e13551fd4f61860742b10e76b54f', 'request_id': 'req-456d0590-dec6-40e2-8910-7aab004129bc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.795968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] The token cache is being invalidate because project 60a1e13551fd4f61860742b10e76b54f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:47.796272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.797013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.800111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1083/2177] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:47 2022] DELETE /identity/v3/projects/60a1e13551fd4f61860742b10e76b54f => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:47.804834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:47.807307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:47.872693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1095/2178] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:47.876678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cf8974-3905-4bc6-ae13-12be086addc7 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:47.877971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-07cf8974-3905-4bc6-ae13-12be086addc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:47.887314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456d0590-dec6-40e2-8910-7aab004129bc admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:47.890419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cf8974-3905-4bc6-ae13-12be086addc7 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:47.891956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:47.892600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:47.892880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:47.893218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] PATH_INFO: `/v3/users/94d849cccfe040319acc25d5a7479729` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:47.894280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] RBAC: Authorizing `identity:delete_user(user_id=94d849cccfe040319acc25d5a7479729)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:47.896856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:47.897055 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:47.897473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:47.897796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:47.898069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] PATH_INFO: `/v3/projects/972e35f87282478ca429b99d33b1e201` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:47.899254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=972e35f87282478ca429b99d33b1e201)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:47.902937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:47.946784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94d849cccfe040319acc25d5a7479729'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.957984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94d849cccfe040319acc25d5a7479729'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.959242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '972e35f87282478ca429b99d33b1e201', 'request_id': 'req-83067998-1911-469b-870c-139ef1980c56'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.959627 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94d849cccfe040319acc25d5a7479729'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.960653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94d849cccfe040319acc25d5a7479729'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.967028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '972e35f87282478ca429b99d33b1e201', 'request_id': 'req-83067998-1911-469b-870c-139ef1980c56'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.968098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f013665a-7f80-483c-ad3c-01168fd36eba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94d849cccfe040319acc25d5a7479729'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.973556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '972e35f87282478ca429b99d33b1e201', 'request_id': 'req-83067998-1911-469b-870c-139ef1980c56'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.990657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] The token cache is being invalidate because project 972e35f87282478ca429b99d33b1e201 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:47.991595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.992280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:47.994006 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1096/2179] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:47 2022] DELETE /identity/v3/projects/972e35f87282478ca429b99d33b1e201 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:47.994914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:47.996186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:47.998596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1084/2180] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:47 2022] DELETE /identity/v3/users/94d849cccfe040319acc25d5a7479729 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:47.999999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245f7278-94ae-4acb-82e6-bf8d2b63d5f2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.000290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245f7278-94ae-4acb-82e6-bf8d2b63d5f2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.000607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245f7278-94ae-4acb-82e6-bf8d2b63d5f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.025023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-245f7278-94ae-4acb-82e6-bf8d2b63d5f2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:48.026350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-245f7278-94ae-4acb-82e6-bf8d2b63d5f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:48.087012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83067998-1911-469b-870c-139ef1980c56 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.093973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:48.108839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1085/2181] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:48.112998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccfb902-2f53-41c2-b0b8-ab1393e1241d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.114194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccfb902-2f53-41c2-b0b8-ab1393e1241d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.114693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccfb902-2f53-41c2-b0b8-ab1393e1241d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.124662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:48.137826 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-eccfb902-2f53-41c2-b0b8-ab1393e1241d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:48.139873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-eccfb902-2f53-41c2-b0b8-ab1393e1241d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:48.154047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1086/2182] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:48.155506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccfb902-2f53-41c2-b0b8-ab1393e1241d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.166636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccfb902-2f53-41c2-b0b8-ab1393e1241d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.167362 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61135171-0cc4-464b-b53d-fc3f3dc6cd72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.168025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61135171-0cc4-464b-b53d-fc3f3dc6cd72 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.168318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61135171-0cc4-464b-b53d-fc3f3dc6cd72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.168727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61135171-0cc4-464b-b53d-fc3f3dc6cd72 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.169381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61135171-0cc4-464b-b53d-fc3f3dc6cd72 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:48.169683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61135171-0cc4-464b-b53d-fc3f3dc6cd72 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.171901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61135171-0cc4-464b-b53d-fc3f3dc6cd72 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.173601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55629c9f-19b5-48d6-ad02-3388bc09d6dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.174692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55629c9f-19b5-48d6-ad02-3388bc09d6dd admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.174811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55629c9f-19b5-48d6-ad02-3388bc09d6dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.175193 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55629c9f-19b5-48d6-ad02-3388bc09d6dd admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.175967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55629c9f-19b5-48d6-ad02-3388bc09d6dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:48.176532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55629c9f-19b5-48d6-ad02-3388bc09d6dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.178966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55629c9f-19b5-48d6-ad02-3388bc09d6dd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.189354 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1097/2183] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:48.191955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61135171-0cc4-464b-b53d-fc3f3dc6cd72 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.195056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1087/2184] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:37:48.199482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55629c9f-19b5-48d6-ad02-3388bc09d6dd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.204199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61135171-0cc4-464b-b53d-fc3f3dc6cd72 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.212011 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe01d87-8042-422f-8ab4-b4d9210829d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.212208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55629c9f-19b5-48d6-ad02-3388bc09d6dd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.212881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe01d87-8042-422f-8ab4-b4d9210829d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.213222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe01d87-8042-422f-8ab4-b4d9210829d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.213651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe01d87-8042-422f-8ab4-b4d9210829d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.214352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe01d87-8042-422f-8ab4-b4d9210829d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.217135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe01d87-8042-422f-8ab4-b4d9210829d8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.220052 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918a752a-b0ed-42f8-8a7a-4baad21b5425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.220952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918a752a-b0ed-42f8-8a7a-4baad21b5425 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.221993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918a752a-b0ed-42f8-8a7a-4baad21b5425 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.222279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918a752a-b0ed-42f8-8a7a-4baad21b5425 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.222978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918a752a-b0ed-42f8-8a7a-4baad21b5425 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.225535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918a752a-b0ed-42f8-8a7a-4baad21b5425 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.249955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1098/2185] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/projects => generated 375 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:48.254293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe01d87-8042-422f-8ab4-b4d9210829d8 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.265301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1088/2186] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/projects => generated 391 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:48.266319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918a752a-b0ed-42f8-8a7a-4baad21b5425 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.330776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe01d87-8042-422f-8ab4-b4d9210829d8 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.344422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918a752a-b0ed-42f8-8a7a-4baad21b5425 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.349995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f74b58e-90da-4db8-adc4-21796f27d0b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.350484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f74b58e-90da-4db8-adc4-21796f27d0b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.350900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f74b58e-90da-4db8-adc4-21796f27d0b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.351178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f74b58e-90da-4db8-adc4-21796f27d0b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.351814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f74b58e-90da-4db8-adc4-21796f27d0b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.353793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f74b58e-90da-4db8-adc4-21796f27d0b3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.378721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1089/2187] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/users => generated 296 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:48.380183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f74b58e-90da-4db8-adc4-21796f27d0b3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.382855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5f74b58e-90da-4db8-adc4-21796f27d0b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:48.394979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f74b58e-90da-4db8-adc4-21796f27d0b3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.399330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1f96ff-cf3e-4acb-b1a7-166f12a25b4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.399924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1f96ff-cf3e-4acb-b1a7-166f12a25b4d admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.400199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1f96ff-cf3e-4acb-b1a7-166f12a25b4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.400441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.400566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1f96ff-cf3e-4acb-b1a7-166f12a25b4d admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.401056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.401157 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1f96ff-cf3e-4acb-b1a7-166f12a25b4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.401336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.401606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] PATH_INFO: `/v3/projects/7b845937294a4feebed33c831de4c77e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.402713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b845937294a4feebed33c831de4c77e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.403131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1f96ff-cf3e-4acb-b1a7-166f12a25b4d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.404629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.424894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1099/2188] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/users => generated 288 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:48.425728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1f96ff-cf3e-4acb-b1a7-166f12a25b4d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.439280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1f96ff-cf3e-4acb-b1a7-166f12a25b4d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.452856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:48.454889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b845937294a4feebed33c831de4c77e', 'request_id': 'req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:48.463567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b845937294a4feebed33c831de4c77e', 'request_id': 'req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:48.471726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b845937294a4feebed33c831de4c77e', 'request_id': 'req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:48.483295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] The token cache is being invalidate because project 7b845937294a4feebed33c831de4c77e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:48.483583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:48.484231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:48.485754 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1090/2189] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:48 2022] DELETE /identity/v3/projects/7b845937294a4feebed33c831de4c77e => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:48.486514 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.487816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:48.503822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149e1686-3b13-4e03-8487-b5d41ddafd8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.504456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e1686-3b13-4e03-8487-b5d41ddafd8b admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.504734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e1686-3b13-4e03-8487-b5d41ddafd8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.505003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149e1686-3b13-4e03-8487-b5d41ddafd8b admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.505740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149e1686-3b13-4e03-8487-b5d41ddafd8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.507368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149e1686-3b13-4e03-8487-b5d41ddafd8b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.516945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1100/2190] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:48 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:48.518120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a43b16-78a9-4648-a733-77d85f99d7c3 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.518403 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a43b16-78a9-4648-a733-77d85f99d7c3 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.518674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a43b16-78a9-4648-a733-77d85f99d7c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.532909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:48.539302 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b5a43b16-78a9-4648-a733-77d85f99d7c3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:48.540477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b5a43b16-78a9-4648-a733-77d85f99d7c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:48.548369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1101/2191] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:48.549134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a43b16-78a9-4648-a733-77d85f99d7c3 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.555981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b5a43b16-78a9-4648-a733-77d85f99d7c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:48.568331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a43b16-78a9-4648-a733-77d85f99d7c3 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.569907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8accc6-3a01-45a8-a9bb-e2d2f5f75f0f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.573834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b1acc3-e5ed-4e4a-9285-a36a395e565a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.574440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b1acc3-e5ed-4e4a-9285-a36a395e565a admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.574827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b1acc3-e5ed-4e4a-9285-a36a395e565a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.575147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b1acc3-e5ed-4e4a-9285-a36a395e565a admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.575296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fca9c30-9f48-4f9a-b595-86ba5c11be49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.575749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b1acc3-e5ed-4e4a-9285-a36a395e565a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.575899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fca9c30-9f48-4f9a-b595-86ba5c11be49 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.576564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fca9c30-9f48-4f9a-b595-86ba5c11be49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.576564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fca9c30-9f48-4f9a-b595-86ba5c11be49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.577144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fca9c30-9f48-4f9a-b595-86ba5c11be49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.577353 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b1acc3-e5ed-4e4a-9285-a36a395e565a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.578850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fca9c30-9f48-4f9a-b595-86ba5c11be49 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.589185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1102/2192] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:48 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:48.590418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cdafa3-a73a-471e-b25c-b15f4489b5fa None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.590673 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cdafa3-a73a-471e-b25c-b15f4489b5fa None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.590947 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cdafa3-a73a-471e-b25c-b15f4489b5fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.603735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1091/2193] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:48 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:48.603735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fca9c30-9f48-4f9a-b595-86ba5c11be49 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.614028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-a2cdafa3-a73a-471e-b25c-b15f4489b5fa None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:48.615272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fca9c30-9f48-4f9a-b595-86ba5c11be49 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.615931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a2cdafa3-a73a-471e-b25c-b15f4489b5fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:48.621963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f2cd6e-9d12-4a9b-9399-e713b3209e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.622639 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f2cd6e-9d12-4a9b-9399-e713b3209e7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.622920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f2cd6e-9d12-4a9b-9399-e713b3209e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.623191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f2cd6e-9d12-4a9b-9399-e713b3209e7e admin admin] PATH_INFO: `/v3/projects/443381f85cba447e8ad8adb79ac4d122/users/0a8b4af8797e4a689d2c6219401b6b87/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.630911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1103/2194] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:48.631997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cdafa3-a73a-471e-b25c-b15f4489b5fa None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.640473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f2cd6e-9d12-4a9b-9399-e713b3209e7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=443381f85cba447e8ad8adb79ac4d122, user_id=0a8b4af8797e4a689d2c6219401b6b87, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.643148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f2cd6e-9d12-4a9b-9399-e713b3209e7e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.647275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cdafa3-a73a-471e-b25c-b15f4489b5fa None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.652700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f499f5e-3e89-4b02-831e-056ca02621b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.653694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f499f5e-3e89-4b02-831e-056ca02621b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.653973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f499f5e-3e89-4b02-831e-056ca02621b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.654241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f499f5e-3e89-4b02-831e-056ca02621b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.655165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f499f5e-3e89-4b02-831e-056ca02621b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.656899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f499f5e-3e89-4b02-831e-056ca02621b7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.657504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1092/2195] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:48 2022] PUT /identity/v3/projects/443381f85cba447e8ad8adb79ac4d122/users/0a8b4af8797e4a689d2c6219401b6b87/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:48.667728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f2cd6e-9d12-4a9b-9399-e713b3209e7e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.675661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1104/2196] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:48 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:48.677370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78ce364-0242-4d2a-9c3b-c8af5977b483 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.677786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78ce364-0242-4d2a-9c3b-c8af5977b483 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.678066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78ce364-0242-4d2a-9c3b-c8af5977b483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.710149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b78ce364-0242-4d2a-9c3b-c8af5977b483 None None] MFA Rules not processed for user `0a8b4af8797e4a689d2c6219401b6b87`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:48.711564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b78ce364-0242-4d2a-9c3b-c8af5977b483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:48.741180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f2cd6e-9d12-4a9b-9399-e713b3209e7e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.747165 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3bfad3-2264-4b80-a121-e852c2645309 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.747845 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3bfad3-2264-4b80-a121-e852c2645309 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.748115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3bfad3-2264-4b80-a121-e852c2645309 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.748410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3bfad3-2264-4b80-a121-e852c2645309 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.749060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3bfad3-2264-4b80-a121-e852c2645309 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:48.749364 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3bfad3-2264-4b80-a121-e852c2645309 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.750974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3bfad3-2264-4b80-a121-e852c2645309 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.769221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1093/2197] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:48.771908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3bfad3-2264-4b80-a121-e852c2645309 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.790778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3bfad3-2264-4b80-a121-e852c2645309 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.791710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1105/2198] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:48.795708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d6ccfd-a1e7-40b1-8d42-bb735f4d4a6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.795842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78ce364-0242-4d2a-9c3b-c8af5977b483 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.796546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d6ccfd-a1e7-40b1-8d42-bb735f4d4a6d admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.796888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d6ccfd-a1e7-40b1-8d42-bb735f4d4a6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.797241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d6ccfd-a1e7-40b1-8d42-bb735f4d4a6d admin admin] PATH_INFO: `/v3/projects/c8e9f882dd924932bd0caa2c124f16a0/users/4e3573c6c40240daaaf79c6fc1fca19a/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.806870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78ce364-0242-4d2a-9c3b-c8af5977b483 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.811635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f5c598-f1ef-44ab-acdb-75c0b199ea25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.812229 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f5c598-f1ef-44ab-acdb-75c0b199ea25 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.812541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f5c598-f1ef-44ab-acdb-75c0b199ea25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.812946 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f5c598-f1ef-44ab-acdb-75c0b199ea25 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.813510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d6ccfd-a1e7-40b1-8d42-bb735f4d4a6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8e9f882dd924932bd0caa2c124f16a0, user_id=4e3573c6c40240daaaf79c6fc1fca19a, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.813618 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f5c598-f1ef-44ab-acdb-75c0b199ea25 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.815468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f5c598-f1ef-44ab-acdb-75c0b199ea25 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.816490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d6ccfd-a1e7-40b1-8d42-bb735f4d4a6d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.832313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1094/2199] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:48 2022] PUT /identity/v3/projects/c8e9f882dd924932bd0caa2c124f16a0/users/4e3573c6c40240daaaf79c6fc1fca19a/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:48.833549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d6ccfd-a1e7-40b1-8d42-bb735f4d4a6d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.865777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1106/2200] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/projects => generated 401 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:48.867908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f5c598-f1ef-44ab-acdb-75c0b199ea25 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:48.912265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d6ccfd-a1e7-40b1-8d42-bb735f4d4a6d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.932206 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:48.935240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:48.949334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f5c598-f1ef-44ab-acdb-75c0b199ea25 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:48.954580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464b79ac-b82d-4ff8-8cb5-2ac113c7f87c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.955188 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464b79ac-b82d-4ff8-8cb5-2ac113c7f87c admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.955468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464b79ac-b82d-4ff8-8cb5-2ac113c7f87c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.955745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464b79ac-b82d-4ff8-8cb5-2ac113c7f87c admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.956401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464b79ac-b82d-4ff8-8cb5-2ac113c7f87c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.958855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464b79ac-b82d-4ff8-8cb5-2ac113c7f87c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.979105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78921fe-9bae-47e7-b51e-6f76c8093a22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:48.979898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78921fe-9bae-47e7-b51e-6f76c8093a22 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.980287 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78921fe-9bae-47e7-b51e-6f76c8093a22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.980578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78921fe-9bae-47e7-b51e-6f76c8093a22 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:48.981230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78921fe-9bae-47e7-b51e-6f76c8093a22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:48.986043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1107/2201] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/users => generated 301 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:48.986188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78921fe-9bae-47e7-b51e-6f76c8093a22 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:48.986271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f385f5-ab20-48a2-940f-c5807df34ed1 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:48.986271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f385f5-ab20-48a2-940f-c5807df34ed1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:48.986271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f385f5-ab20-48a2-940f-c5807df34ed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.016172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1095/2202] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/projects => generated 393 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:49.016557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-a6f385f5-ab20-48a2-940f-c5807df34ed1 None None] MFA Rules not processed for user `4e3573c6c40240daaaf79c6fc1fca19a`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:49.017470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78921fe-9bae-47e7-b51e-6f76c8093a22 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.018401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a6f385f5-ab20-48a2-940f-c5807df34ed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:49.095081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78921fe-9bae-47e7-b51e-6f76c8093a22 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.100721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7108c72c-398b-4de3-939d-7f941b269531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.101323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7108c72c-398b-4de3-939d-7f941b269531 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.101604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7108c72c-398b-4de3-939d-7f941b269531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.102001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7108c72c-398b-4de3-939d-7f941b269531 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.102660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7108c72c-398b-4de3-939d-7f941b269531 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.104268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7108c72c-398b-4de3-939d-7f941b269531 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.109875 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1108/2203] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:49.111375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f385f5-ab20-48a2-940f-c5807df34ed1 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.118650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1096/2204] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:49 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:49.118650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7108c72c-398b-4de3-939d-7f941b269531 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.122627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f385f5-ab20-48a2-940f-c5807df34ed1 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.127375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac8dd62-2639-44fd-9148-e500745c0c4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.127970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac8dd62-2639-44fd-9148-e500745c0c4e admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.128247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac8dd62-2639-44fd-9148-e500745c0c4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.128363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7108c72c-398b-4de3-939d-7f941b269531 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.128621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac8dd62-2639-44fd-9148-e500745c0c4e admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.129248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac8dd62-2639-44fd-9148-e500745c0c4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.131163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac8dd62-2639-44fd-9148-e500745c0c4e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.133112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a212fa14-4f3a-41e5-b265-de978980347a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.133690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a212fa14-4f3a-41e5-b265-de978980347a admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.133970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a212fa14-4f3a-41e5-b265-de978980347a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.134243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a212fa14-4f3a-41e5-b265-de978980347a admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.134865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a212fa14-4f3a-41e5-b265-de978980347a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.136809 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a212fa14-4f3a-41e5-b265-de978980347a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.157106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1109/2205] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:49 2022] POST /identity/v3/users => generated 303 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:49.158330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac8dd62-2639-44fd-9148-e500745c0c4e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.168718 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac8dd62-2639-44fd-9148-e500745c0c4e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.172754 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1097/2206] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:49 2022] POST /identity/v3/projects => generated 375 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:49.172754 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a212fa14-4f3a-41e5-b265-de978980347a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.234099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e91dc19-50bc-4edb-ad8c-3b11c325e413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.234718 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e91dc19-50bc-4edb-ad8c-3b11c325e413 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.235003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e91dc19-50bc-4edb-ad8c-3b11c325e413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.235276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e91dc19-50bc-4edb-ad8c-3b11c325e413 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.235932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e91dc19-50bc-4edb-ad8c-3b11c325e413 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.237613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e91dc19-50bc-4edb-ad8c-3b11c325e413 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.247234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a212fa14-4f3a-41e5-b265-de978980347a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.248323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1110/2207] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:49 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:49.249178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e91dc19-50bc-4edb-ad8c-3b11c325e413 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.252659 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cffae26-f1e1-4adf-8ee8-58161bbe6b5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.253293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cffae26-f1e1-4adf-8ee8-58161bbe6b5d admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.253573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cffae26-f1e1-4adf-8ee8-58161bbe6b5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.253854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cffae26-f1e1-4adf-8ee8-58161bbe6b5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.254729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cffae26-f1e1-4adf-8ee8-58161bbe6b5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.257065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cffae26-f1e1-4adf-8ee8-58161bbe6b5d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.258730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e91dc19-50bc-4edb-ad8c-3b11c325e413 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.264902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ab6562-9d77-4966-a3db-93346c32947e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.265717 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab6562-9d77-4966-a3db-93346c32947e admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.266057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab6562-9d77-4966-a3db-93346c32947e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.266479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab6562-9d77-4966-a3db-93346c32947e admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.267149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ab6562-9d77-4966-a3db-93346c32947e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.267394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1098/2208] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:49 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:49.268365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cffae26-f1e1-4adf-8ee8-58161bbe6b5d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.270151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ab6562-9d77-4966-a3db-93346c32947e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.279473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cffae26-f1e1-4adf-8ee8-58161bbe6b5d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.284476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc5e46d-14cc-45ff-a585-d72ab58c88fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.285149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc5e46d-14cc-45ff-a585-d72ab58c88fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.285465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc5e46d-14cc-45ff-a585-d72ab58c88fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.285785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc5e46d-14cc-45ff-a585-d72ab58c88fd admin admin] PATH_INFO: `/v3/projects/d5f2c9b71d5146ed8979a19fbc86981e/users/be2535c595cc4dc4bbb84dd3f17adaeb/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.292873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1111/2209] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:49 2022] POST /identity/v3/users => generated 288 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:49.293682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ab6562-9d77-4966-a3db-93346c32947e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.302673 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc5e46d-14cc-45ff-a585-d72ab58c88fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5f2c9b71d5146ed8979a19fbc86981e, user_id=be2535c595cc4dc4bbb84dd3f17adaeb, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.303642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ab6562-9d77-4966-a3db-93346c32947e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.305521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc5e46d-14cc-45ff-a585-d72ab58c88fd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.308253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c1f5f8-3354-46a5-8f6f-245a218affc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.308686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:49.308915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c1f5f8-3354-46a5-8f6f-245a218affc1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.309201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c1f5f8-3354-46a5-8f6f-245a218affc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.309506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c1f5f8-3354-46a5-8f6f-245a218affc1 admin admin] PATH_INFO: `/v3/projects/eb2a9671332c49cea516e25271b80e6a/users/0679396c1a7c4687a08d331d547ada78/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.321735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1099/2210] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:49 2022] PUT /identity/v3/projects/d5f2c9b71d5146ed8979a19fbc86981e/users/be2535c595cc4dc4bbb84dd3f17adaeb/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:49.323253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc5e46d-14cc-45ff-a585-d72ab58c88fd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.327799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c1f5f8-3354-46a5-8f6f-245a218affc1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb2a9671332c49cea516e25271b80e6a, user_id=0679396c1a7c4687a08d331d547ada78, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.329537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c1f5f8-3354-46a5-8f6f-245a218affc1 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.344063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1112/2211] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:49 2022] PUT /identity/v3/projects/eb2a9671332c49cea516e25271b80e6a/users/0679396c1a7c4687a08d331d547ada78/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:49.345733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6df714-624a-4af6-bc55-9525a5ac8cd8 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.346225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6df714-624a-4af6-bc55-9525a5ac8cd8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.346698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6df714-624a-4af6-bc55-9525a5ac8cd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.354777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:49.377796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-8c6df714-624a-4af6-bc55-9525a5ac8cd8 None None] MFA Rules not processed for user `be2535c595cc4dc4bbb84dd3f17adaeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:49.379038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8c6df714-624a-4af6-bc55-9525a5ac8cd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:49.396266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc5e46d-14cc-45ff-a585-d72ab58c88fd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.448490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1113/2212] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:49 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:49.453015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6df714-624a-4af6-bc55-9525a5ac8cd8 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.469025 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27a45bd-d28f-4adf-9fe8-84c618278332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.469629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27a45bd-d28f-4adf-9fe8-84c618278332 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.469915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27a45bd-d28f-4adf-9fe8-84c618278332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.470192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27a45bd-d28f-4adf-9fe8-84c618278332 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.470840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27a45bd-d28f-4adf-9fe8-84c618278332 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.472491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27a45bd-d28f-4adf-9fe8-84c618278332 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.486072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1100/2213] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:49 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:49.486072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27a45bd-d28f-4adf-9fe8-84c618278332 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.494744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27a45bd-d28f-4adf-9fe8-84c618278332 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.499479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5143a5-37c2-42a7-918a-c5073e1792ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.500075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5143a5-37c2-42a7-918a-c5073e1792ac admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.500368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5143a5-37c2-42a7-918a-c5073e1792ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.500664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5143a5-37c2-42a7-918a-c5073e1792ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.501295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5143a5-37c2-42a7-918a-c5073e1792ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.503208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5143a5-37c2-42a7-918a-c5073e1792ac admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.520156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6df714-624a-4af6-bc55-9525a5ac8cd8 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.528298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb7f680-33f1-440a-af44-64aefede8672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.528954 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb7f680-33f1-440a-af44-64aefede8672 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.529230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb7f680-33f1-440a-af44-64aefede8672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.529496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb7f680-33f1-440a-af44-64aefede8672 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.530145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb7f680-33f1-440a-af44-64aefede8672 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.531755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb7f680-33f1-440a-af44-64aefede8672 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.537063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1101/2214] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:49 2022] POST /identity/v3/projects => generated 401 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:49.537063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5143a5-37c2-42a7-918a-c5073e1792ac admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.545984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1114/2215] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:49 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:49.546800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb7f680-33f1-440a-af44-64aefede8672 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.558149 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb7f680-33f1-440a-af44-64aefede8672 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.607178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5143a5-37c2-42a7-918a-c5073e1792ac admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.612516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7981fed2-829d-4cf5-8457-6063bca21fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.613120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7981fed2-829d-4cf5-8457-6063bca21fd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.613393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7981fed2-829d-4cf5-8457-6063bca21fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.613661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7981fed2-829d-4cf5-8457-6063bca21fd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.614313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7981fed2-829d-4cf5-8457-6063bca21fd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.616059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7981fed2-829d-4cf5-8457-6063bca21fd8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.620032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0568faf9-981e-47f7-bda8-e7692ea670e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.621815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0568faf9-981e-47f7-bda8-e7692ea670e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.621815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0568faf9-981e-47f7-bda8-e7692ea670e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.621815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0568faf9-981e-47f7-bda8-e7692ea670e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.621815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0568faf9-981e-47f7-bda8-e7692ea670e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.623704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0568faf9-981e-47f7-bda8-e7692ea670e5 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.627126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1102/2216] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:49 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:49.628137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7981fed2-829d-4cf5-8457-6063bca21fd8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.641844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7981fed2-829d-4cf5-8457-6063bca21fd8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.649577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627e97c6-43a2-41b7-beb2-e759bb21998c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.650452 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627e97c6-43a2-41b7-beb2-e759bb21998c admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.650785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627e97c6-43a2-41b7-beb2-e759bb21998c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.651094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627e97c6-43a2-41b7-beb2-e759bb21998c admin admin] PATH_INFO: `/v3/projects/eb2a9671332c49cea516e25271b80e6a/users/0679396c1a7c4687a08d331d547ada78/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.653078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627e97c6-43a2-41b7-beb2-e759bb21998c admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb2a9671332c49cea516e25271b80e6a, user_id=0679396c1a7c4687a08d331d547ada78, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.656500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1115/2217] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:49 2022] POST /identity/v3/users => generated 307 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:49.656500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0568faf9-981e-47f7-bda8-e7692ea670e5 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.656912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627e97c6-43a2-41b7-beb2-e759bb21998c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.663397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:49.667477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0568faf9-981e-47f7-bda8-e7692ea670e5 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.669049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1103/2218] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:49 2022] PUT /identity/v3/projects/eb2a9671332c49cea516e25271b80e6a/users/0679396c1a7c4687a08d331d547ada78/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:49.670173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627e97c6-43a2-41b7-beb2-e759bb21998c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.681537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:49.734907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76d5cb1-1321-49c9-8e6f-d79339e90f30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.735591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76d5cb1-1321-49c9-8e6f-d79339e90f30 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.735858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76d5cb1-1321-49c9-8e6f-d79339e90f30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.736130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76d5cb1-1321-49c9-8e6f-d79339e90f30 admin admin] PATH_INFO: `/v3/projects/db31d191b6674c21ab8baf4309e57d08/users/21469d849d8f44f1a91a699dd3e3d75e/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.738619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627e97c6-43a2-41b7-beb2-e759bb21998c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.747162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b3c961-6604-44ee-a254-d71bee44e1d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.747778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b3c961-6604-44ee-a254-d71bee44e1d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.748056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b3c961-6604-44ee-a254-d71bee44e1d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.748332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b3c961-6604-44ee-a254-d71bee44e1d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.749003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b3c961-6604-44ee-a254-d71bee44e1d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.750618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b3c961-6604-44ee-a254-d71bee44e1d1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.752144 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76d5cb1-1321-49c9-8e6f-d79339e90f30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db31d191b6674c21ab8baf4309e57d08, user_id=21469d849d8f44f1a91a699dd3e3d75e, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.754820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76d5cb1-1321-49c9-8e6f-d79339e90f30 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.761439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1104/2219] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:49 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:49.763174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d519f27-47ce-42dc-9d3f-4b5f59d79db0 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.763951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d519f27-47ce-42dc-9d3f-4b5f59d79db0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.764474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d519f27-47ce-42dc-9d3f-4b5f59d79db0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.768921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1116/2220] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:49 2022] PUT /identity/v3/projects/db31d191b6674c21ab8baf4309e57d08/users/21469d849d8f44f1a91a699dd3e3d75e/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:49.769849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76d5cb1-1321-49c9-8e6f-d79339e90f30 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.796337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4d519f27-47ce-42dc-9d3f-4b5f59d79db0 None None] MFA Rules not processed for user `0679396c1a7c4687a08d331d547ada78`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:49.797625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4d519f27-47ce-42dc-9d3f-4b5f59d79db0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:49.870385 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76d5cb1-1321-49c9-8e6f-d79339e90f30 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:49.878444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b619e0-4577-4942-9b0d-cced60d8b260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:49.879563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b619e0-4577-4942-9b0d-cced60d8b260 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.880067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b619e0-4577-4942-9b0d-cced60d8b260 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.880598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b619e0-4577-4942-9b0d-cced60d8b260 admin admin] PATH_INFO: `/v3/projects/22252ae1ef1d46c4aaba016c35747dc2/users/472bfa3799bf42f6b5ed6827885f66c9/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.893089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1105/2221] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:49 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:49.894921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac536fe-2532-465f-82fb-9489505b5573 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:49.895209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac536fe-2532-465f-82fb-9489505b5573 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:49.895483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac536fe-2532-465f-82fb-9489505b5573 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:49.920146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b619e0-4577-4942-9b0d-cced60d8b260 admin admin] RBAC: Authorizing `identity:create_grant(project_id=22252ae1ef1d46c4aaba016c35747dc2, user_id=472bfa3799bf42f6b5ed6827885f66c9, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:49.924172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b619e0-4577-4942-9b0d-cced60d8b260 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:49.931865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9ac536fe-2532-465f-82fb-9489505b5573 None None] MFA Rules not processed for user `21469d849d8f44f1a91a699dd3e3d75e`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:49.933318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9ac536fe-2532-465f-82fb-9489505b5573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:49.943222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1117/2222] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:49 2022] PUT /identity/v3/projects/22252ae1ef1d46c4aaba016c35747dc2/users/472bfa3799bf42f6b5ed6827885f66c9/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:49.944217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b619e0-4577-4942-9b0d-cced60d8b260 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:49.946740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-56b619e0-4577-4942-9b0d-cced60d8b260 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:50.025929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b619e0-4577-4942-9b0d-cced60d8b260 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.026350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1106/2223] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:49 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:50.027200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac536fe-2532-465f-82fb-9489505b5573 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.032090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a86c5a1-79e7-4def-9743-4bde82982ff4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.032801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a86c5a1-79e7-4def-9743-4bde82982ff4 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.033081 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a86c5a1-79e7-4def-9743-4bde82982ff4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.033353 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a86c5a1-79e7-4def-9743-4bde82982ff4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.044482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:50.082327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:50.119359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a86c5a1-79e7-4def-9743-4bde82982ff4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.121399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a86c5a1-79e7-4def-9743-4bde82982ff4 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.121968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac536fe-2532-465f-82fb-9489505b5573 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.126646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60db553-aa44-4676-a51f-3f9c5c044924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.127255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60db553-aa44-4676-a51f-3f9c5c044924 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.127552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60db553-aa44-4676-a51f-3f9c5c044924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.127821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60db553-aa44-4676-a51f-3f9c5c044924 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.128532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60db553-aa44-4676-a51f-3f9c5c044924 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.130255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60db553-aa44-4676-a51f-3f9c5c044924 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.131955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1118/2224] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:49 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:50.132783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a86c5a1-79e7-4def-9743-4bde82982ff4 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.140634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1107/2225] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:50 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:50.143629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a86c5a1-79e7-4def-9743-4bde82982ff4 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.145041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60db553-aa44-4676-a51f-3f9c5c044924 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.149374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a36ef2-004a-4d90-954b-8d2bd7bf4144 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.150238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a36ef2-004a-4d90-954b-8d2bd7bf4144 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.150516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a36ef2-004a-4d90-954b-8d2bd7bf4144 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.150783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a36ef2-004a-4d90-954b-8d2bd7bf4144 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.152119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a36ef2-004a-4d90-954b-8d2bd7bf4144 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.155037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a36ef2-004a-4d90-954b-8d2bd7bf4144 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.161007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60db553-aa44-4676-a51f-3f9c5c044924 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.166859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483d8b91-6c50-4c21-95d6-1bafe625f058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.167784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483d8b91-6c50-4c21-95d6-1bafe625f058 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.168070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483d8b91-6c50-4c21-95d6-1bafe625f058 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.168363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483d8b91-6c50-4c21-95d6-1bafe625f058 admin admin] PATH_INFO: `/v3/projects/22252ae1ef1d46c4aaba016c35747dc2/users/472bfa3799bf42f6b5ed6827885f66c9/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.170405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483d8b91-6c50-4c21-95d6-1bafe625f058 admin admin] RBAC: Authorizing `identity:create_grant(project_id=22252ae1ef1d46c4aaba016c35747dc2, user_id=472bfa3799bf42f6b5ed6827885f66c9, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.173183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483d8b91-6c50-4c21-95d6-1bafe625f058 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.185475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1108/2226] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:50 2022] PUT /identity/v3/projects/22252ae1ef1d46c4aaba016c35747dc2/users/472bfa3799bf42f6b5ed6827885f66c9/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:50.189983 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1119/2227] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:50 2022] POST /identity/v3/projects => generated 375 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:50.193876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483d8b91-6c50-4c21-95d6-1bafe625f058 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.194163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a36ef2-004a-4d90-954b-8d2bd7bf4144 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.196018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-483d8b91-6c50-4c21-95d6-1bafe625f058 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:50.256563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483d8b91-6c50-4c21-95d6-1bafe625f058 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.264591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a36ef2-004a-4d90-954b-8d2bd7bf4144 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.269317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35058cf-9c71-4641-a2b8-38b7c70d16f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.269914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35058cf-9c71-4641-a2b8-38b7c70d16f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.270189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35058cf-9c71-4641-a2b8-38b7c70d16f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.270461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35058cf-9c71-4641-a2b8-38b7c70d16f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.271426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35058cf-9c71-4641-a2b8-38b7c70d16f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.273386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35058cf-9c71-4641-a2b8-38b7c70d16f8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.299743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1120/2228] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:50 2022] POST /identity/v3/users => generated 294 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:50.301492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7938ceab-d9d6-4e32-8c8a-782c56e532ac None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.301774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7938ceab-d9d6-4e32-8c8a-782c56e532ac None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.302294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7938ceab-d9d6-4e32-8c8a-782c56e532ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.311951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c82c6c1-2556-4571-a680-e8fb208c8be8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.312598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c82c6c1-2556-4571-a680-e8fb208c8be8 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.312874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c82c6c1-2556-4571-a680-e8fb208c8be8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.313154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c82c6c1-2556-4571-a680-e8fb208c8be8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.331016 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7938ceab-d9d6-4e32-8c8a-782c56e532ac None None] MFA Rules not processed for user `472bfa3799bf42f6b5ed6827885f66c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:50.332236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7938ceab-d9d6-4e32-8c8a-782c56e532ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:50.377245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c82c6c1-2556-4571-a680-e8fb208c8be8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.378992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c82c6c1-2556-4571-a680-e8fb208c8be8 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.389035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1109/2229] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:50 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:50.389970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c82c6c1-2556-4571-a680-e8fb208c8be8 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.397215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:50.400977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c82c6c1-2556-4571-a680-e8fb208c8be8 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.406681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d62497a-40d7-4c7b-930e-54500d7a7363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.407307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d62497a-40d7-4c7b-930e-54500d7a7363 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.407588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d62497a-40d7-4c7b-930e-54500d7a7363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.407863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d62497a-40d7-4c7b-930e-54500d7a7363 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.408536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d62497a-40d7-4c7b-930e-54500d7a7363 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.410145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d62497a-40d7-4c7b-930e-54500d7a7363 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.415289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1121/2230] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:50 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:50.425439 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1110/2231] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:50 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:50.426415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7938ceab-d9d6-4e32-8c8a-782c56e532ac None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.434971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d62497a-40d7-4c7b-930e-54500d7a7363 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.441110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7938ceab-d9d6-4e32-8c8a-782c56e532ac None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.446293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d62497a-40d7-4c7b-930e-54500d7a7363 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.446897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9025c8-e4a7-4764-92d4-577585b6a762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.447585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9025c8-e4a7-4764-92d4-577585b6a762 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.447868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9025c8-e4a7-4764-92d4-577585b6a762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.448186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9025c8-e4a7-4764-92d4-577585b6a762 admin admin] PATH_INFO: `/v3/projects/2573036b95b2422c9c53975d1c29ebde/users/36d95de6283d4190afe4f1ac6e8478d9/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.450712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e6433-063b-4e6f-9b2f-3e3c412672fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.452051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e6433-063b-4e6f-9b2f-3e3c412672fe service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.452051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e6433-063b-4e6f-9b2f-3e3c412672fe service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.452051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e6433-063b-4e6f-9b2f-3e3c412672fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.462625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e6433-063b-4e6f-9b2f-3e3c412672fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.464595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e6433-063b-4e6f-9b2f-3e3c412672fe service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.465739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9025c8-e4a7-4764-92d4-577585b6a762 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2573036b95b2422c9c53975d1c29ebde, user_id=36d95de6283d4190afe4f1ac6e8478d9, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.468713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9025c8-e4a7-4764-92d4-577585b6a762 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.475228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1111/2232] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:50 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:50.484278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1122/2233] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:50 2022] PUT /identity/v3/projects/2573036b95b2422c9c53975d1c29ebde/users/36d95de6283d4190afe4f1ac6e8478d9/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:50.488792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e6433-063b-4e6f-9b2f-3e3c412672fe service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.500949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e6433-063b-4e6f-9b2f-3e3c412672fe service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.576666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9025c8-e4a7-4764-92d4-577585b6a762 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.582338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:50.584527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb67ce5-a17c-4a19-8191-470efc0a89db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.585316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb67ce5-a17c-4a19-8191-470efc0a89db admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.585574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb67ce5-a17c-4a19-8191-470efc0a89db admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.585852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb67ce5-a17c-4a19-8191-470efc0a89db admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.586561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb67ce5-a17c-4a19-8191-470efc0a89db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.588392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb67ce5-a17c-4a19-8191-470efc0a89db admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.602467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1112/2234] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:50 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:50.603562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb67ce5-a17c-4a19-8191-470efc0a89db admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.620018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb67ce5-a17c-4a19-8191-470efc0a89db admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.626208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d03731-021b-40ad-b4e1-bb1a568e8e2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.626874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d03731-021b-40ad-b4e1-bb1a568e8e2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.627158 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d03731-021b-40ad-b4e1-bb1a568e8e2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.627499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d03731-021b-40ad-b4e1-bb1a568e8e2c admin admin] PATH_INFO: `/v3/projects/2573036b95b2422c9c53975d1c29ebde/users/36d95de6283d4190afe4f1ac6e8478d9/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.630441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d03731-021b-40ad-b4e1-bb1a568e8e2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2573036b95b2422c9c53975d1c29ebde, user_id=36d95de6283d4190afe4f1ac6e8478d9, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.633742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d03731-021b-40ad-b4e1-bb1a568e8e2c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.646435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9025c8-e4a7-4764-92d4-577585b6a762 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.655889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1113/2235] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:50 2022] PUT /identity/v3/projects/2573036b95b2422c9c53975d1c29ebde/users/36d95de6283d4190afe4f1ac6e8478d9/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:50.657903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d6265-8ff7-49b1-b1a3-2744e15a1f7e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.658299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d6265-8ff7-49b1-b1a3-2744e15a1f7e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.658581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d6265-8ff7-49b1-b1a3-2744e15a1f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.704762 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-799d6265-8ff7-49b1-b1a3-2744e15a1f7e None None] MFA Rules not processed for user `36d95de6283d4190afe4f1ac6e8478d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:50.706471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-799d6265-8ff7-49b1-b1a3-2744e15a1f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:50.735063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfddfdf-28ae-4a0c-9ef8-f6e83e7cd67d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.735851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfddfdf-28ae-4a0c-9ef8-f6e83e7cd67d service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.736182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfddfdf-28ae-4a0c-9ef8-f6e83e7cd67d service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.736486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfddfdf-28ae-4a0c-9ef8-f6e83e7cd67d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.807739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1114/2236] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:50 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:37:50.821438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d6265-8ff7-49b1-b1a3-2744e15a1f7e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:50.823292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfddfdf-28ae-4a0c-9ef8-f6e83e7cd67d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.824889 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfddfdf-28ae-4a0c-9ef8-f6e83e7cd67d service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.831692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d6265-8ff7-49b1-b1a3-2744e15a1f7e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:50.835082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1123/2237] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:50 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:37:50.836929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31506d2d-354b-49f2-ad6d-a72b8dbf43fa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:50.837643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31506d2d-354b-49f2-ad6d-a72b8dbf43fa service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:50.838031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31506d2d-354b-49f2-ad6d-a72b8dbf43fa service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:50.838315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31506d2d-354b-49f2-ad6d-a72b8dbf43fa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:50.891121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:50.904538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31506d2d-354b-49f2-ad6d-a72b8dbf43fa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:50.905984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31506d2d-354b-49f2-ad6d-a72b8dbf43fa service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:50.915337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1115/2238] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:50 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:37:52.711994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfddfdf-28ae-4a0c-9ef8-f6e83e7cd67d service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:52.720785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfddfdf-28ae-4a0c-9ef8-f6e83e7cd67d service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:52.724585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ecaf5c-2078-4bad-aa27-a2fa031c3262 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:52.725192 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ecaf5c-2078-4bad-aa27-a2fa031c3262 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:52.725472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ecaf5c-2078-4bad-aa27-a2fa031c3262 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:52.725748 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ecaf5c-2078-4bad-aa27-a2fa031c3262 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:52.787798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ecaf5c-2078-4bad-aa27-a2fa031c3262 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:52.789272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ecaf5c-2078-4bad-aa27-a2fa031c3262 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:52.799054 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1124/2239] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:52 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:53.381739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31506d2d-354b-49f2-ad6d-a72b8dbf43fa service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:53.390434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31506d2d-354b-49f2-ad6d-a72b8dbf43fa service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:53.394063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2baecfc-8745-413f-a21e-6c87704b2e67 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:53.394654 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2baecfc-8745-413f-a21e-6c87704b2e67 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:53.394932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2baecfc-8745-413f-a21e-6c87704b2e67 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:53.395204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2baecfc-8745-413f-a21e-6c87704b2e67 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:53.404321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2baecfc-8745-413f-a21e-6c87704b2e67 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:53.405775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2baecfc-8745-413f-a21e-6c87704b2e67 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:53.416266 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1116/2240] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:53 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:53.944645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ecaf5c-2078-4bad-aa27-a2fa031c3262 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:53.962287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:54.024758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ecaf5c-2078-4bad-aa27-a2fa031c3262 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:54.029257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:54.029879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:54.030163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:54.030437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] PATH_INFO: `/v3/users/0a8b4af8797e4a689d2c6219401b6b87` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:54.031563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a8b4af8797e4a689d2c6219401b6b87)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:54.033557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:54.080602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a8b4af8797e4a689d2c6219401b6b87'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.088443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a8b4af8797e4a689d2c6219401b6b87'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.089062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a8b4af8797e4a689d2c6219401b6b87'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.089674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a8b4af8797e4a689d2c6219401b6b87'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.101545 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a8b4af8797e4a689d2c6219401b6b87'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.114010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1125/2241] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:53 2022] DELETE /identity/v3/users/0a8b4af8797e4a689d2c6219401b6b87 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:54.118641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053217ac-d089-4f52-9cfd-1589469764b4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:54.118965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053217ac-d089-4f52-9cfd-1589469764b4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:54.119247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053217ac-d089-4f52-9cfd-1589469764b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:54.138931 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-053217ac-d089-4f52-9cfd-1589469764b4 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:54.140280 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-053217ac-d089-4f52-9cfd-1589469764b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:54.200625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1117/2242] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:54.211407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:54.212879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:54.273185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff82740-8ffe-48f7-95c2-31c65ef17f01 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:54.277061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fdc97d-6053-4b90-a372-dcd84e42f047 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:54.277610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fdc97d-6053-4b90-a372-dcd84e42f047 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:54.277887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fdc97d-6053-4b90-a372-dcd84e42f047 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:54.278156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fdc97d-6053-4b90-a372-dcd84e42f047 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:54.287662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fdc97d-6053-4b90-a372-dcd84e42f047 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:54.289123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fdc97d-6053-4b90-a372-dcd84e42f047 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:54.298428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1126/2243] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:54.575608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053217ac-d089-4f52-9cfd-1589469764b4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:54.577107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-053217ac-d089-4f52-9cfd-1589469764b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:54.589709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053217ac-d089-4f52-9cfd-1589469764b4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:54.595044 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:54.595682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:54.596554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:54.597015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] PATH_INFO: `/v3/projects/443381f85cba447e8ad8adb79ac4d122` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:54.603859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] RBAC: Authorizing `identity:delete_project(project_id=443381f85cba447e8ad8adb79ac4d122)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:54.606036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:54.659889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '443381f85cba447e8ad8adb79ac4d122', 'request_id': 'req-259fc488-4a41-40e7-8a20-1cec6f860196'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.668591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '443381f85cba447e8ad8adb79ac4d122', 'request_id': 'req-259fc488-4a41-40e7-8a20-1cec6f860196'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.677526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '443381f85cba447e8ad8adb79ac4d122', 'request_id': 'req-259fc488-4a41-40e7-8a20-1cec6f860196'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.689049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] The token cache is being invalidate because project 443381f85cba447e8ad8adb79ac4d122 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:54.689512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.690468 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.691876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1118/2244] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:54 2022] DELETE /identity/v3/projects/443381f85cba447e8ad8adb79ac4d122 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:54.696946 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fdc97d-6053-4b90-a372-dcd84e42f047 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:54.708177 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fdc97d-6053-4b90-a372-dcd84e42f047 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:54.750368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:54.789728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:54.790366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:54.790662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:54.790927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] PATH_INFO: `/v3/users/0679396c1a7c4687a08d331d547ada78` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:54.792067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0679396c1a7c4687a08d331d547ada78)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:54.794168 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:54.897105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0679396c1a7c4687a08d331d547ada78'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.906832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0679396c1a7c4687a08d331d547ada78'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.907460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0679396c1a7c4687a08d331d547ada78'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.908095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0679396c1a7c4687a08d331d547ada78'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.924000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-62c86175-a484-44df-9eca-a3336ca32161 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0679396c1a7c4687a08d331d547ada78'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:54.943037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1127/2245] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:54 2022] DELETE /identity/v3/users/0679396c1a7c4687a08d331d547ada78 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:55.229616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:55.231126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:55.247000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259fc488-4a41-40e7-8a20-1cec6f860196 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:55.298882 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:55.306793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:55.307473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:55.307936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:55.308536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] PATH_INFO: `/v3/projects/eb2a9671332c49cea516e25271b80e6a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:55.309775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb2a9671332c49cea516e25271b80e6a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:55.311702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:55.358335 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb2a9671332c49cea516e25271b80e6a', 'request_id': 'req-2554ad1f-0b8f-4270-8ab4-4044e553d695'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:55.364949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb2a9671332c49cea516e25271b80e6a', 'request_id': 'req-2554ad1f-0b8f-4270-8ab4-4044e553d695'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:55.372703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb2a9671332c49cea516e25271b80e6a', 'request_id': 'req-2554ad1f-0b8f-4270-8ab4-4044e553d695'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:55.383500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] The token cache is being invalidate because project eb2a9671332c49cea516e25271b80e6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:37:55.383820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:55.384500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2554ad1f-0b8f-4270-8ab4-4044e553d695 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:55.385994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1119/2246] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:37:55 2022] DELETE /identity/v3/projects/eb2a9671332c49cea516e25271b80e6a => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:55.390846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa70a63-d9be-4f46-9a06-7074af242f17 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:55.391304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa70a63-d9be-4f46-9a06-7074af242f17 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:55.391395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa70a63-d9be-4f46-9a06-7074af242f17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:55.410660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-daa70a63-d9be-4f46-9a06-7074af242f17 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:55.412408 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-daa70a63-d9be-4f46-9a06-7074af242f17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:55.469786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1128/2247] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:55.478077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c495c9-2c5f-4a5c-bdbb-f82bded7a74b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:55.478363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c495c9-2c5f-4a5c-bdbb-f82bded7a74b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:55.478640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c495c9-2c5f-4a5c-bdbb-f82bded7a74b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:55.499320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-04c495c9-2c5f-4a5c-bdbb-f82bded7a74b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:55.500531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-04c495c9-2c5f-4a5c-bdbb-f82bded7a74b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:55.508553 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1120/2248] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:55.512762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa70a63-d9be-4f46-9a06-7074af242f17 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:55.522552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa70a63-d9be-4f46-9a06-7074af242f17 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:55.527209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a92801-f97f-4028-841e-418e2a483147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:55.527803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a92801-f97f-4028-841e-418e2a483147 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:55.528082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a92801-f97f-4028-841e-418e2a483147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:55.528372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a92801-f97f-4028-841e-418e2a483147 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:55.529027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a92801-f97f-4028-841e-418e2a483147 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:37:55.529335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a92801-f97f-4028-841e-418e2a483147 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:55.530931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a92801-f97f-4028-841e-418e2a483147 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:55.542924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1129/2249] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:37:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:37:55.547095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c495c9-2c5f-4a5c-bdbb-f82bded7a74b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:55.557183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c495c9-2c5f-4a5c-bdbb-f82bded7a74b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:55.561689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2acbb6-9a2b-4d71-ab83-4bd4f5fa45e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:55.562273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2acbb6-9a2b-4d71-ab83-4bd4f5fa45e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:55.562549 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2acbb6-9a2b-4d71-ab83-4bd4f5fa45e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:55.562818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2acbb6-9a2b-4d71-ab83-4bd4f5fa45e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:55.563445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2acbb6-9a2b-4d71-ab83-4bd4f5fa45e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:55.565370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2acbb6-9a2b-4d71-ab83-4bd4f5fa45e5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:55.597267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1121/2250] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:55 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:55.601049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a92801-f97f-4028-841e-418e2a483147 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:55.610840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a92801-f97f-4028-841e-418e2a483147 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:55.649592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:55.671068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a46ff82-06e1-4c49-b887-3dab3a639189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:55.671668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a46ff82-06e1-4c49-b887-3dab3a639189 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:55.671945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a46ff82-06e1-4c49-b887-3dab3a639189 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:55.672216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a46ff82-06e1-4c49-b887-3dab3a639189 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:55.672884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a46ff82-06e1-4c49-b887-3dab3a639189 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:55.674774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a46ff82-06e1-4c49-b887-3dab3a639189 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:55.695287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1130/2251] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:55 2022] POST /identity/v3/users => generated 292 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:37:55.699816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2acbb6-9a2b-4d71-ab83-4bd4f5fa45e5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:55.710461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2acbb6-9a2b-4d71-ab83-4bd4f5fa45e5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:55.715024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53e3be5-4f19-47c2-8d28-c8edf2c0a8c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:55.715609 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53e3be5-4f19-47c2-8d28-c8edf2c0a8c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:55.715887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53e3be5-4f19-47c2-8d28-c8edf2c0a8c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:55.716157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53e3be5-4f19-47c2-8d28-c8edf2c0a8c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:55.716838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53e3be5-4f19-47c2-8d28-c8edf2c0a8c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:55.718422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53e3be5-4f19-47c2-8d28-c8edf2c0a8c0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:55.728945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1122/2252] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:55.732724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a46ff82-06e1-4c49-b887-3dab3a639189 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:55.742805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a46ff82-06e1-4c49-b887-3dab3a639189 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:55.747335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f19071-7af1-4461-a9ed-0e293ac726a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:55.747913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f19071-7af1-4461-a9ed-0e293ac726a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:55.748190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f19071-7af1-4461-a9ed-0e293ac726a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:55.748493 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f19071-7af1-4461-a9ed-0e293ac726a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:55.749152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f19071-7af1-4461-a9ed-0e293ac726a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:55.750721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f19071-7af1-4461-a9ed-0e293ac726a7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:55.760740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1131/2253] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:55 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:37:55.764694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53e3be5-4f19-47c2-8d28-c8edf2c0a8c0 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:55.776087 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53e3be5-4f19-47c2-8d28-c8edf2c0a8c0 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:55.780687 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2858ca4-8b62-4ed7-8dbf-2f0f8247a8e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:55.781318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2858ca4-8b62-4ed7-8dbf-2f0f8247a8e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:55.781593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2858ca4-8b62-4ed7-8dbf-2f0f8247a8e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:55.781866 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2858ca4-8b62-4ed7-8dbf-2f0f8247a8e3 admin admin] PATH_INFO: `/v3/projects/ff33aea8cc614b16bdcd5392504bf09f/users/5669ef04249a4a80842951e2f80cd1af/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:55.796922 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2858ca4-8b62-4ed7-8dbf-2f0f8247a8e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff33aea8cc614b16bdcd5392504bf09f, user_id=5669ef04249a4a80842951e2f80cd1af, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:55.799568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2858ca4-8b62-4ed7-8dbf-2f0f8247a8e3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:55.812212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1123/2254] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:55 2022] PUT /identity/v3/projects/ff33aea8cc614b16bdcd5392504bf09f/users/5669ef04249a4a80842951e2f80cd1af/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:55.822626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d936da-0004-48da-8eed-e3c7c40048f1 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:55.822913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d936da-0004-48da-8eed-e3c7c40048f1 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:55.823183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d936da-0004-48da-8eed-e3c7c40048f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:55.853451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-36d936da-0004-48da-8eed-e3c7c40048f1 None None] MFA Rules not processed for user `5669ef04249a4a80842951e2f80cd1af`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:55.854703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-36d936da-0004-48da-8eed-e3c7c40048f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:55.918676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1132/2255] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:55 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:55.923077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2858ca4-8b62-4ed7-8dbf-2f0f8247a8e3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:55.997697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2858ca4-8b62-4ed7-8dbf-2f0f8247a8e3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:56.002300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a62b1-7729-4e3d-9945-929b4efb741a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:56.002952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a62b1-7729-4e3d-9945-929b4efb741a admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:56.003276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a62b1-7729-4e3d-9945-929b4efb741a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:56.003600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a62b1-7729-4e3d-9945-929b4efb741a admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:56.004235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64a62b1-7729-4e3d-9945-929b4efb741a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:56.007145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64a62b1-7729-4e3d-9945-929b4efb741a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:56.045920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1124/2256] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:37:55 2022] POST /identity/v3/projects => generated 385 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:56.049905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d936da-0004-48da-8eed-e3c7c40048f1 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:56.070577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d936da-0004-48da-8eed-e3c7c40048f1 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:56.147578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b942abf9-1395-4c1b-a0fe-f76d13a13fc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:56.148180 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b942abf9-1395-4c1b-a0fe-f76d13a13fc7 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:56.148483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b942abf9-1395-4c1b-a0fe-f76d13a13fc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:56.148756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b942abf9-1395-4c1b-a0fe-f76d13a13fc7 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:56.149374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b942abf9-1395-4c1b-a0fe-f76d13a13fc7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:56.151266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b942abf9-1395-4c1b-a0fe-f76d13a13fc7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:56.171950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1133/2257] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:37:56 2022] POST /identity/v3/users => generated 299 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:37:56.175963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a62b1-7729-4e3d-9945-929b4efb741a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:56.186405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a62b1-7729-4e3d-9945-929b4efb741a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:56.191073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041ebecc-cdd1-49d5-97d2-a7b0129a3825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:56.191661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041ebecc-cdd1-49d5-97d2-a7b0129a3825 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:56.191945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041ebecc-cdd1-49d5-97d2-a7b0129a3825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:56.192767 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041ebecc-cdd1-49d5-97d2-a7b0129a3825 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:56.193411 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041ebecc-cdd1-49d5-97d2-a7b0129a3825 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:56.195003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041ebecc-cdd1-49d5-97d2-a7b0129a3825 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:56.197304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:56.211030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1125/2258] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:56 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:56.212161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b942abf9-1395-4c1b-a0fe-f76d13a13fc7 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:56.222138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b942abf9-1395-4c1b-a0fe-f76d13a13fc7 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:56.227270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d082e-d71e-4b1d-9280-b8b1c4cc7db8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:56.227910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d082e-d71e-4b1d-9280-b8b1c4cc7db8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:56.228186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d082e-d71e-4b1d-9280-b8b1c4cc7db8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:56.228492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d082e-d71e-4b1d-9280-b8b1c4cc7db8 admin admin] PATH_INFO: `/v3/projects/263619296b544313949d1e183fbc348e/users/27a20294e95e4e10b8a24b23193ec654/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:56.243631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323d082e-d71e-4b1d-9280-b8b1c4cc7db8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=263619296b544313949d1e183fbc348e, user_id=27a20294e95e4e10b8a24b23193ec654, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:56.246343 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323d082e-d71e-4b1d-9280-b8b1c4cc7db8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:56.260079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1134/2259] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:56 2022] PUT /identity/v3/projects/263619296b544313949d1e183fbc348e/users/27a20294e95e4e10b8a24b23193ec654/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:37:56.264214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041ebecc-cdd1-49d5-97d2-a7b0129a3825 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:56.275186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041ebecc-cdd1-49d5-97d2-a7b0129a3825 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:56.337871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03050b5-ac74-4c2b-904a-6301a355ab42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:56.338384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03050b5-ac74-4c2b-904a-6301a355ab42 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:56.338663 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03050b5-ac74-4c2b-904a-6301a355ab42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:56.338938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03050b5-ac74-4c2b-904a-6301a355ab42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:56.339584 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03050b5-ac74-4c2b-904a-6301a355ab42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:56.341316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03050b5-ac74-4c2b-904a-6301a355ab42 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:56.351328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1126/2260] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:37:56 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:37:56.355726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d082e-d71e-4b1d-9280-b8b1c4cc7db8 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:56.366187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d082e-d71e-4b1d-9280-b8b1c4cc7db8 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:56.370711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b867a6-edcd-4674-9e25-b0d2e312e9c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:56.371344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b867a6-edcd-4674-9e25-b0d2e312e9c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:56.371620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b867a6-edcd-4674-9e25-b0d2e312e9c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:56.371891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b867a6-edcd-4674-9e25-b0d2e312e9c6 admin admin] PATH_INFO: `/v3/projects/263619296b544313949d1e183fbc348e/users/27a20294e95e4e10b8a24b23193ec654/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:56.373795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b867a6-edcd-4674-9e25-b0d2e312e9c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=263619296b544313949d1e183fbc348e, user_id=27a20294e95e4e10b8a24b23193ec654, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:56.376434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b867a6-edcd-4674-9e25-b0d2e312e9c6 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:56.383176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:56.387806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1135/2261] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:37:56 2022] PUT /identity/v3/projects/263619296b544313949d1e183fbc348e/users/27a20294e95e4e10b8a24b23193ec654/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:56.400127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9660f4-ef14-45e6-8f4e-3e8e548a3ea6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:56.400453 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9660f4-ef14-45e6-8f4e-3e8e548a3ea6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:56.400732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9660f4-ef14-45e6-8f4e-3e8e548a3ea6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:56.428212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6f9660f4-ef14-45e6-8f4e-3e8e548a3ea6 None None] MFA Rules not processed for user `27a20294e95e4e10b8a24b23193ec654`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:56.429452 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6f9660f4-ef14-45e6-8f4e-3e8e548a3ea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:56.497295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1127/2262] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:56 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:56.508620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b867a6-edcd-4674-9e25-b0d2e312e9c6 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:56.510320 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-04b867a6-edcd-4674-9e25-b0d2e312e9c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:56.567771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b867a6-edcd-4674-9e25-b0d2e312e9c6 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:56.571718 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e9f8c2-99fd-432e-a38c-fea4aba1a7ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:56.572586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9f8c2-99fd-432e-a38c-fea4aba1a7ae service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:56.572859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9f8c2-99fd-432e-a38c-fea4aba1a7ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:56.573130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9f8c2-99fd-432e-a38c-fea4aba1a7ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:56.627905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e9f8c2-99fd-432e-a38c-fea4aba1a7ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:56.629289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e9f8c2-99fd-432e-a38c-fea4aba1a7ae service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:56.638953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1136/2263] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:56 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:37:59.574979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9660f4-ef14-45e6-8f4e-3e8e548a3ea6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:59.576481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6f9660f4-ef14-45e6-8f4e-3e8e548a3ea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:59.662031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9660f4-ef14-45e6-8f4e-3e8e548a3ea6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:59.666602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:59.667257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:59.667577 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:59.667817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] PATH_INFO: `/v3/users/be2535c595cc4dc4bbb84dd3f17adaeb` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:59.668955 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] RBAC: Authorizing `identity:delete_user(user_id=be2535c595cc4dc4bbb84dd3f17adaeb)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:59.670878 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:59.693850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:37:59.725966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be2535c595cc4dc4bbb84dd3f17adaeb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:59.733394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be2535c595cc4dc4bbb84dd3f17adaeb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:59.733995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be2535c595cc4dc4bbb84dd3f17adaeb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:59.734605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be2535c595cc4dc4bbb84dd3f17adaeb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:59.742371 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be2535c595cc4dc4bbb84dd3f17adaeb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:37:59.760968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1128/2264] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:37:59 2022] DELETE /identity/v3/users/be2535c595cc4dc4bbb84dd3f17adaeb => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:37:59.765374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b4ebfe-0779-4d19-bb5a-629315bed96e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:59.765823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b4ebfe-0779-4d19-bb5a-629315bed96e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:59.765932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b4ebfe-0779-4d19-bb5a-629315bed96e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:59.785507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-12b4ebfe-0779-4d19-bb5a-629315bed96e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:37:59.787359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-12b4ebfe-0779-4d19-bb5a-629315bed96e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:59.850794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1137/2265] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:37:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:37:59.861433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:37:59.863119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:37:59.922544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d3347c-00d9-4449-b9a1-999b2633acaf admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:37:59.926481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feff68db-07b1-4254-8f76-96459bb87eda service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:37:59.927086 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feff68db-07b1-4254-8f76-96459bb87eda service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:37:59.927356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feff68db-07b1-4254-8f76-96459bb87eda service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:37:59.927632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feff68db-07b1-4254-8f76-96459bb87eda service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:37:59.937347 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feff68db-07b1-4254-8f76-96459bb87eda service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:37:59.938854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feff68db-07b1-4254-8f76-96459bb87eda service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:37:59.950113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1129/2266] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:37:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:38:00.257333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b4ebfe-0779-4d19-bb5a-629315bed96e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:00.258973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-12b4ebfe-0779-4d19-bb5a-629315bed96e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:00.275752 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b4ebfe-0779-4d19-bb5a-629315bed96e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:00.281604 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:00.282249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:00.282677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:00.282964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] PATH_INFO: `/v3/projects/d5f2c9b71d5146ed8979a19fbc86981e` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:00.284492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] RBAC: Authorizing `identity:delete_project(project_id=d5f2c9b71d5146ed8979a19fbc86981e)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:00.288195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:00.356969 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd5f2c9b71d5146ed8979a19fbc86981e', 'request_id': 'req-5c5babca-3009-4e67-8527-bff04e600cbe'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.358430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:00.364440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd5f2c9b71d5146ed8979a19fbc86981e', 'request_id': 'req-5c5babca-3009-4e67-8527-bff04e600cbe'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.372685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd5f2c9b71d5146ed8979a19fbc86981e', 'request_id': 'req-5c5babca-3009-4e67-8527-bff04e600cbe'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.386734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] The token cache is being invalidate because project d5f2c9b71d5146ed8979a19fbc86981e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:00.387030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.387705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.389440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1138/2267] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:00 2022] DELETE /identity/v3/projects/d5f2c9b71d5146ed8979a19fbc86981e => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:00.393488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feff68db-07b1-4254-8f76-96459bb87eda service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:00.404080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feff68db-07b1-4254-8f76-96459bb87eda service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:00.492116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:00.492977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:00.493263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:00.493590 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] PATH_INFO: `/v3/users/472bfa3799bf42f6b5ed6827885f66c9` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:00.494647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] RBAC: Authorizing `identity:delete_user(user_id=472bfa3799bf42f6b5ed6827885f66c9)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:00.496778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:00.558208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:00.560665 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '472bfa3799bf42f6b5ed6827885f66c9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.570024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '472bfa3799bf42f6b5ed6827885f66c9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.571038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '472bfa3799bf42f6b5ed6827885f66c9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.572356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '472bfa3799bf42f6b5ed6827885f66c9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.579530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-445645a7-7a20-477e-bbbe-815d183c3427 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '472bfa3799bf42f6b5ed6827885f66c9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.608603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1130/2268] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:00 2022] DELETE /identity/v3/users/472bfa3799bf42f6b5ed6827885f66c9 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:00.769811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:00.772971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:00.796824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5babca-3009-4e67-8527-bff04e600cbe admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:00.892639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:00.893633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:00.894169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:00.894784 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] PATH_INFO: `/v3/users/4e3573c6c40240daaaf79c6fc1fca19a` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:00.896884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e3573c6c40240daaaf79c6fc1fca19a)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:00.899009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:00.964834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e3573c6c40240daaaf79c6fc1fca19a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.977743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e3573c6c40240daaaf79c6fc1fca19a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.978546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e3573c6c40240daaaf79c6fc1fca19a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.980026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e3573c6c40240daaaf79c6fc1fca19a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:00.998824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e3573c6c40240daaaf79c6fc1fca19a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.024560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1139/2269] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:00 2022] DELETE /identity/v3/users/4e3573c6c40240daaaf79c6fc1fca19a => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:01.029329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36e09c0-57b9-489a-8b03-ad3c7dcc7167 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.029607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36e09c0-57b9-489a-8b03-ad3c7dcc7167 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.030067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36e09c0-57b9-489a-8b03-ad3c7dcc7167 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.056944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f36e09c0-57b9-489a-8b03-ad3c7dcc7167 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:01.058205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f36e09c0-57b9-489a-8b03-ad3c7dcc7167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:01.064267 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.066021 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:01.129474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1131/2270] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:01.138639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:01.139785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36e09c0-57b9-489a-8b03-ad3c7dcc7167 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.141082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f36e09c0-57b9-489a-8b03-ad3c7dcc7167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:01.142261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f471ce3-4f4f-456d-adce-896b95a60470 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:01.149264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:01.150045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.150479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.150821 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] PATH_INFO: `/v3/projects/22252ae1ef1d46c4aaba016c35747dc2` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.152515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] RBAC: Authorizing `identity:delete_project(project_id=22252ae1ef1d46c4aaba016c35747dc2)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:01.155315 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:01.204061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36e09c0-57b9-489a-8b03-ad3c7dcc7167 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:01.206512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '22252ae1ef1d46c4aaba016c35747dc2', 'request_id': 'req-daef03a1-601a-465b-9867-7243079d006e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.213915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '22252ae1ef1d46c4aaba016c35747dc2', 'request_id': 'req-daef03a1-601a-465b-9867-7243079d006e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.219926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '22252ae1ef1d46c4aaba016c35747dc2', 'request_id': 'req-daef03a1-601a-465b-9867-7243079d006e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.236963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] The token cache is being invalidate because project 22252ae1ef1d46c4aaba016c35747dc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:01.237258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.237906 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-daef03a1-601a-465b-9867-7243079d006e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.240145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1140/2271] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:01 2022] DELETE /identity/v3/projects/22252ae1ef1d46c4aaba016c35747dc2 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:38:01.244626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997db5cb-0401-44e6-b70e-aef15a51bb01 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.244910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997db5cb-0401-44e6-b70e-aef15a51bb01 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.245193 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997db5cb-0401-44e6-b70e-aef15a51bb01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.257981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b2ec4a-23d8-4ec4-b469-8d821baa6677 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:01.258727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b2ec4a-23d8-4ec4-b469-8d821baa6677 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.259013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b2ec4a-23d8-4ec4-b469-8d821baa6677 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.259313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b2ec4a-23d8-4ec4-b469-8d821baa6677 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.261589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a6b2ec4a-23d8-4ec4-b469-8d821baa6677 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:01.265271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-997db5cb-0401-44e6-b70e-aef15a51bb01 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:01.266463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-997db5cb-0401-44e6-b70e-aef15a51bb01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:01.294002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:01.330899 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1141/2272] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:38:01.338695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729351dd-e6ea-4768-8b7d-1bc51e64fd63 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.339033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729351dd-e6ea-4768-8b7d-1bc51e64fd63 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.339258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729351dd-e6ea-4768-8b7d-1bc51e64fd63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.346822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b2ec4a-23d8-4ec4-b469-8d821baa6677 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:01.350118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b2ec4a-23d8-4ec4-b469-8d821baa6677 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:01.359374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-729351dd-e6ea-4768-8b7d-1bc51e64fd63 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:01.360989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-729351dd-e6ea-4768-8b7d-1bc51e64fd63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:01.369385 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1132/2273] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:38:01.371421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1142/2274] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:38:01.376010 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b2ec4a-23d8-4ec4-b469-8d821baa6677 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.387622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b2ec4a-23d8-4ec4-b469-8d821baa6677 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:01.396814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ed2bce-ec61-40e9-a082-c54e640b280b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:01.397564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ed2bce-ec61-40e9-a082-c54e640b280b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.398094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ed2bce-ec61-40e9-a082-c54e640b280b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.398471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ed2bce-ec61-40e9-a082-c54e640b280b admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.399621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ed2bce-ec61-40e9-a082-c54e640b280b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:38:01.400374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ed2bce-ec61-40e9-a082-c54e640b280b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:01.402107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ed2bce-ec61-40e9-a082-c54e640b280b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:01.415972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1133/2275] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:38:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:38:01.420914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729351dd-e6ea-4768-8b7d-1bc51e64fd63 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.432536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729351dd-e6ea-4768-8b7d-1bc51e64fd63 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:01.437297 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560c7e6d-8bc5-4446-8e47-738a254c975c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:01.437908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560c7e6d-8bc5-4446-8e47-738a254c975c admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.438183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560c7e6d-8bc5-4446-8e47-738a254c975c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.438455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560c7e6d-8bc5-4446-8e47-738a254c975c admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.439092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560c7e6d-8bc5-4446-8e47-738a254c975c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:01.441071 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560c7e6d-8bc5-4446-8e47-738a254c975c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:01.473341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1143/2276] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:38:01 2022] POST /identity/v3/projects => generated 375 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:01.477899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ed2bce-ec61-40e9-a082-c54e640b280b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.488240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ed2bce-ec61-40e9-a082-c54e640b280b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:01.564983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848a6a93-0159-48ea-8c14-a98827c1893b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:01.564983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848a6a93-0159-48ea-8c14-a98827c1893b admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.564983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848a6a93-0159-48ea-8c14-a98827c1893b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.564983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848a6a93-0159-48ea-8c14-a98827c1893b admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.565497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848a6a93-0159-48ea-8c14-a98827c1893b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:01.569321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848a6a93-0159-48ea-8c14-a98827c1893b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:01.602333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1134/2277] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:38:01 2022] POST /identity/v3/users => generated 288 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:01.606970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560c7e6d-8bc5-4446-8e47-738a254c975c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.624395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560c7e6d-8bc5-4446-8e47-738a254c975c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:01.632553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bb192d-6817-49fa-889a-f5780fe5f807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:01.633238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bb192d-6817-49fa-889a-f5780fe5f807 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.633519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bb192d-6817-49fa-889a-f5780fe5f807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.633786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bb192d-6817-49fa-889a-f5780fe5f807 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.634779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bb192d-6817-49fa-889a-f5780fe5f807 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:01.637186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bb192d-6817-49fa-889a-f5780fe5f807 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:01.658084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1144/2278] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:01 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:01.662336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848a6a93-0159-48ea-8c14-a98827c1893b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.675184 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848a6a93-0159-48ea-8c14-a98827c1893b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:01.684509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9492b6-c3eb-4575-8752-da61e28e9a97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:01.684509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9492b6-c3eb-4575-8752-da61e28e9a97 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.684509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9492b6-c3eb-4575-8752-da61e28e9a97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.684509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9492b6-c3eb-4575-8752-da61e28e9a97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.685056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9492b6-c3eb-4575-8752-da61e28e9a97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:01.686166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9492b6-c3eb-4575-8752-da61e28e9a97 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:01.699586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1135/2279] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:01 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:01.704969 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bb192d-6817-49fa-889a-f5780fe5f807 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.719059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9492b6-c3eb-4575-8752-da61e28e9a97 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.720733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bb192d-6817-49fa-889a-f5780fe5f807 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:01.723605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ee9492b6-c3eb-4575-8752-da61e28e9a97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:01.735009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47b277f-b461-45de-b2dd-259fd5c8c2a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:01.735803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b277f-b461-45de-b2dd-259fd5c8c2a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.736156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b277f-b461-45de-b2dd-259fd5c8c2a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.736731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b277f-b461-45de-b2dd-259fd5c8c2a7 admin admin] PATH_INFO: `/v3/projects/eddc4331819c4814ad83f4d8bcf17d21/users/93f891bbe4dc44cd835eda441f6cf1dd/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.756398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9492b6-c3eb-4575-8752-da61e28e9a97 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:01.763360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:01.764502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.765083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.765201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47b277f-b461-45de-b2dd-259fd5c8c2a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eddc4331819c4814ad83f4d8bcf17d21, user_id=93f891bbe4dc44cd835eda441f6cf1dd, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:01.765730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] PATH_INFO: `/v3/projects/c8e9f882dd924932bd0caa2c124f16a0` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.768093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8e9f882dd924932bd0caa2c124f16a0)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:01.768499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47b277f-b461-45de-b2dd-259fd5c8c2a7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:01.772325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:01.791380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1145/2280] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:38:01 2022] PUT /identity/v3/projects/eddc4331819c4814ad83f4d8bcf17d21/users/93f891bbe4dc44cd835eda441f6cf1dd/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:01.805962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4876a81-489f-4a6f-9844-919bab40b249 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.806241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4876a81-489f-4a6f-9844-919bab40b249 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.811644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4876a81-489f-4a6f-9844-919bab40b249 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:01.840816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8e9f882dd924932bd0caa2c124f16a0', 'request_id': 'req-4c72d03c-f488-4703-9728-4a06fd8373a4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.847633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8e9f882dd924932bd0caa2c124f16a0', 'request_id': 'req-4c72d03c-f488-4703-9728-4a06fd8373a4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.856624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-f4876a81-489f-4a6f-9844-919bab40b249 None None] MFA Rules not processed for user `93f891bbe4dc44cd835eda441f6cf1dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:01.858080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f4876a81-489f-4a6f-9844-919bab40b249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:01.863419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8e9f882dd924932bd0caa2c124f16a0', 'request_id': 'req-4c72d03c-f488-4703-9728-4a06fd8373a4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.878822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:01.881586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] The token cache is being invalidate because project c8e9f882dd924932bd0caa2c124f16a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:01.882214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.882831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:01.886216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1136/2281] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:01 2022] DELETE /identity/v3/projects/c8e9f882dd924932bd0caa2c124f16a0 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:38:01.890292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.891799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:01.914194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:01.954693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1146/2282] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:01.969730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4876a81-489f-4a6f-9844-919bab40b249 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:01.971487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f4876a81-489f-4a6f-9844-919bab40b249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:01.990341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c72d03c-f488-4703-9728-4a06fd8373a4 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:01.997138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:01.997969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:01.998426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:01.998704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] PATH_INFO: `/v3/users/21469d849d8f44f1a91a699dd3e3d75e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:02.000446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] RBAC: Authorizing `identity:delete_user(user_id=21469d849d8f44f1a91a699dd3e3d75e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:02.004399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:02.097629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4876a81-489f-4a6f-9844-919bab40b249 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:02.102847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dbc385-e0e1-43f6-8ac2-665379857cb3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e07b6890d3004dd78faed48a1ab90086', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:02.103513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dbc385-e0e1-43f6-8ac2-665379857cb3 service glance] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:02.103798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dbc385-e0e1-43f6-8ac2-665379857cb3 service glance] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:02.104073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dbc385-e0e1-43f6-8ac2-665379857cb3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:02.112625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21469d849d8f44f1a91a699dd3e3d75e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.114211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dbc385-e0e1-43f6-8ac2-665379857cb3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:02.116528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dbc385-e0e1-43f6-8ac2-665379857cb3 service glance] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:02.129806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21469d849d8f44f1a91a699dd3e3d75e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.131005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21469d849d8f44f1a91a699dd3e3d75e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.131832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21469d849d8f44f1a91a699dd3e3d75e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.134080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1147/2283] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:01 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:38:02.149004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21469d849d8f44f1a91a699dd3e3d75e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.174232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dbc385-e0e1-43f6-8ac2-665379857cb3 service glance] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:02.175971 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e9dbc385-e0e1-43f6-8ac2-665379857cb3 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:02.187405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1137/2284] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:01 2022] DELETE /identity/v3/users/21469d849d8f44f1a91a699dd3e3d75e => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:38:02.253704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dbc385-e0e1-43f6-8ac2-665379857cb3 service glance] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:02.354961 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679ac079-1692-4a81-82d5-e1fea66fb51d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:02.355749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679ac079-1692-4a81-82d5-e1fea66fb51d service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:02.356424 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679ac079-1692-4a81-82d5-e1fea66fb51d service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:02.356690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679ac079-1692-4a81-82d5-e1fea66fb51d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:02.359492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-679ac079-1692-4a81-82d5-e1fea66fb51d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:02.530664 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:02.538279 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:02.539913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:02.580310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679ac079-1692-4a81-82d5-e1fea66fb51d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:02.582529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679ac079-1692-4a81-82d5-e1fea66fb51d service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:02.596078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1148/2285] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:38:02.627083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ced80-d737-4d03-a3d7-641db056410d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:02.633077 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:02.633573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:02.633859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:02.634136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] PATH_INFO: `/v3/projects/db31d191b6674c21ab8baf4309e57d08` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:02.635282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db31d191b6674c21ab8baf4309e57d08)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:02.637932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:02.691649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:02.706643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db31d191b6674c21ab8baf4309e57d08', 'request_id': 'req-92fd80ca-661c-4b6d-bda9-532779394987'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.714582 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db31d191b6674c21ab8baf4309e57d08', 'request_id': 'req-92fd80ca-661c-4b6d-bda9-532779394987'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.740199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db31d191b6674c21ab8baf4309e57d08', 'request_id': 'req-92fd80ca-661c-4b6d-bda9-532779394987'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.754892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] The token cache is being invalidate because project db31d191b6674c21ab8baf4309e57d08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:02.756272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.757621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.759749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1138/2286] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:02 2022] DELETE /identity/v3/projects/db31d191b6674c21ab8baf4309e57d08 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:38:02.764724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679ac079-1692-4a81-82d5-e1fea66fb51d service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:02.783535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679ac079-1692-4a81-82d5-e1fea66fb51d service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:02.851390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:02.852026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:02.852367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:02.852631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] PATH_INFO: `/v3/users/36d95de6283d4190afe4f1ac6e8478d9` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:02.853765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] RBAC: Authorizing `identity:delete_user(user_id=36d95de6283d4190afe4f1ac6e8478d9)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:02.855710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:02.914846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36d95de6283d4190afe4f1ac6e8478d9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.924960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36d95de6283d4190afe4f1ac6e8478d9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.926597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36d95de6283d4190afe4f1ac6e8478d9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.927391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36d95de6283d4190afe4f1ac6e8478d9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.941997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-65693f1b-6dfa-4cb7-93c1-28a21cf68e8b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36d95de6283d4190afe4f1ac6e8478d9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:02.972392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1149/2287] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:02 2022] DELETE /identity/v3/users/36d95de6283d4190afe4f1ac6e8478d9 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:03.248083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:03.249551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:03.266162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fd80ca-661c-4b6d-bda9-532779394987 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:03.325924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:03.326540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:03.326814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:03.327088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] PATH_INFO: `/v3/projects/2573036b95b2422c9c53975d1c29ebde` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:03.328216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2573036b95b2422c9c53975d1c29ebde)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:03.330173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:03.376928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2573036b95b2422c9c53975d1c29ebde', 'request_id': 'req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:03.383298 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2573036b95b2422c9c53975d1c29ebde', 'request_id': 'req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:03.391209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2573036b95b2422c9c53975d1c29ebde', 'request_id': 'req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:03.403308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] The token cache is being invalidate because project 2573036b95b2422c9c53975d1c29ebde was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:03.403597 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:03.404241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ee4de5b-8380-4cf0-8e5d-937e91534c2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:03.405777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1139/2288] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:03 2022] DELETE /identity/v3/projects/2573036b95b2422c9c53975d1c29ebde => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:03.410620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92f6a81-5ecb-465d-a12c-728b3250490e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:03.410905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92f6a81-5ecb-465d-a12c-728b3250490e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:03.411178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92f6a81-5ecb-465d-a12c-728b3250490e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:03.431094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e92f6a81-5ecb-465d-a12c-728b3250490e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:03.432298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e92f6a81-5ecb-465d-a12c-728b3250490e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:03.473328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:03.497945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1150/2289] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:03.505807 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70430c8-e390-49bf-94ff-7ca1355936dc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:03.506168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70430c8-e390-49bf-94ff-7ca1355936dc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:03.506504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70430c8-e390-49bf-94ff-7ca1355936dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:03.527154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b70430c8-e390-49bf-94ff-7ca1355936dc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:03.528366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b70430c8-e390-49bf-94ff-7ca1355936dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:03.536886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1140/2290] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:03.541166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92f6a81-5ecb-465d-a12c-728b3250490e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:03.550818 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92f6a81-5ecb-465d-a12c-728b3250490e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:03.555515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875aa870-5a1d-4a04-afc5-18817ae11d03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:03.556105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875aa870-5a1d-4a04-afc5-18817ae11d03 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:03.556414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875aa870-5a1d-4a04-afc5-18817ae11d03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:03.556687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875aa870-5a1d-4a04-afc5-18817ae11d03 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:03.557335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875aa870-5a1d-4a04-afc5-18817ae11d03 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:38:03.557636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875aa870-5a1d-4a04-afc5-18817ae11d03 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:03.559213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875aa870-5a1d-4a04-afc5-18817ae11d03 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:03.571192 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1151/2291] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:38:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:38:03.575605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70430c8-e390-49bf-94ff-7ca1355936dc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:03.585581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70430c8-e390-49bf-94ff-7ca1355936dc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:03.590130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914510c7-051b-45cf-ada4-aebad10a317c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:03.590707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914510c7-051b-45cf-ada4-aebad10a317c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:03.590983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914510c7-051b-45cf-ada4-aebad10a317c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:03.591252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914510c7-051b-45cf-ada4-aebad10a317c admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:03.591871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914510c7-051b-45cf-ada4-aebad10a317c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:03.593788 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914510c7-051b-45cf-ada4-aebad10a317c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:03.620998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1141/2292] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:38:03 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:03.625136 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875aa870-5a1d-4a04-afc5-18817ae11d03 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:03.635300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875aa870-5a1d-4a04-afc5-18817ae11d03 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:03.692078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d21aae6-77b3-44d0-ab07-ab0a92537c01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:03.692730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d21aae6-77b3-44d0-ab07-ab0a92537c01 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:03.693036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d21aae6-77b3-44d0-ab07-ab0a92537c01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:03.693329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d21aae6-77b3-44d0-ab07-ab0a92537c01 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:03.693973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d21aae6-77b3-44d0-ab07-ab0a92537c01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:03.695858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d21aae6-77b3-44d0-ab07-ab0a92537c01 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:03.716455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1152/2293] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:38:03 2022] POST /identity/v3/users => generated 292 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:03.720529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914510c7-051b-45cf-ada4-aebad10a317c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:03.731260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914510c7-051b-45cf-ada4-aebad10a317c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:03.736069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8439a7f-aa2d-45eb-8425-4cbe5e52bf61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:03.736691 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8439a7f-aa2d-45eb-8425-4cbe5e52bf61 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:03.736965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8439a7f-aa2d-45eb-8425-4cbe5e52bf61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:03.737237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8439a7f-aa2d-45eb-8425-4cbe5e52bf61 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:03.737879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8439a7f-aa2d-45eb-8425-4cbe5e52bf61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:03.740107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8439a7f-aa2d-45eb-8425-4cbe5e52bf61 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:03.750457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1142/2294] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:03.754360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d21aae6-77b3-44d0-ab07-ab0a92537c01 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:03.765653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d21aae6-77b3-44d0-ab07-ab0a92537c01 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:03.771695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d078b61-0650-4dcd-93a7-f2e33045bb28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:03.772397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d078b61-0650-4dcd-93a7-f2e33045bb28 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:03.772682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d078b61-0650-4dcd-93a7-f2e33045bb28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:03.773044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d078b61-0650-4dcd-93a7-f2e33045bb28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:03.773625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d078b61-0650-4dcd-93a7-f2e33045bb28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:03.775176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d078b61-0650-4dcd-93a7-f2e33045bb28 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:03.785579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1153/2295] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:03.790081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8439a7f-aa2d-45eb-8425-4cbe5e52bf61 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:03.800172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8439a7f-aa2d-45eb-8425-4cbe5e52bf61 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:03.805704 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6abf1d-0b1c-4813-bb75-60ae1cf3b592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:03.806351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6abf1d-0b1c-4813-bb75-60ae1cf3b592 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:03.806627 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6abf1d-0b1c-4813-bb75-60ae1cf3b592 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:03.806900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6abf1d-0b1c-4813-bb75-60ae1cf3b592 admin admin] PATH_INFO: `/v3/projects/a2fac30e93fc422bb7375ca123b3b307/users/b4744313d73940898b2b79e997d79b7f/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:03.819333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:03.823444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6abf1d-0b1c-4813-bb75-60ae1cf3b592 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2fac30e93fc422bb7375ca123b3b307, user_id=b4744313d73940898b2b79e997d79b7f, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:03.826773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6abf1d-0b1c-4813-bb75-60ae1cf3b592 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:03.841943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1143/2296] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:38:03 2022] PUT /identity/v3/projects/a2fac30e93fc422bb7375ca123b3b307/users/b4744313d73940898b2b79e997d79b7f/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:03.852438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d33b7f-c0ec-4c96-aad5-a30472fd71c8 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:03.852907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d33b7f-c0ec-4c96-aad5-a30472fd71c8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:03.853016 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d33b7f-c0ec-4c96-aad5-a30472fd71c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:03.882263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-23d33b7f-c0ec-4c96-aad5-a30472fd71c8 None None] MFA Rules not processed for user `b4744313d73940898b2b79e997d79b7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:03.883500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-23d33b7f-c0ec-4c96-aad5-a30472fd71c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:03.948124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1154/2297] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:03 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:03.952584 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6abf1d-0b1c-4813-bb75-60ae1cf3b592 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.016076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6abf1d-0b1c-4813-bb75-60ae1cf3b592 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.020964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c254f2-f097-4f61-9824-2debe5a6957b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.021560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c254f2-f097-4f61-9824-2debe5a6957b admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.021837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c254f2-f097-4f61-9824-2debe5a6957b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.022112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c254f2-f097-4f61-9824-2debe5a6957b admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.022752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c254f2-f097-4f61-9824-2debe5a6957b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.024666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c254f2-f097-4f61-9824-2debe5a6957b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.053317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1144/2298] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:38:03 2022] POST /identity/v3/projects => generated 385 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:38:04.057372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d33b7f-c0ec-4c96-aad5-a30472fd71c8 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.068574 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d33b7f-c0ec-4c96-aad5-a30472fd71c8 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.102810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:04.128004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcb51eb-ac2f-4b23-8994-1b77dc57a3cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.128638 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcb51eb-ac2f-4b23-8994-1b77dc57a3cd admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.128919 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcb51eb-ac2f-4b23-8994-1b77dc57a3cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.129189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcb51eb-ac2f-4b23-8994-1b77dc57a3cd admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.129810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcb51eb-ac2f-4b23-8994-1b77dc57a3cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.132375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcb51eb-ac2f-4b23-8994-1b77dc57a3cd admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.153737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1155/2299] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:38:04 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:04.157638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c254f2-f097-4f61-9824-2debe5a6957b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.168537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c254f2-f097-4f61-9824-2debe5a6957b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.173184 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37989679-2189-4527-9e7b-b0881138450b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.173791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37989679-2189-4527-9e7b-b0881138450b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.174057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37989679-2189-4527-9e7b-b0881138450b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.174331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37989679-2189-4527-9e7b-b0881138450b admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.174964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37989679-2189-4527-9e7b-b0881138450b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.176598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37989679-2189-4527-9e7b-b0881138450b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.187042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1145/2300] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:04.190810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcb51eb-ac2f-4b23-8994-1b77dc57a3cd admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.201079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcb51eb-ac2f-4b23-8994-1b77dc57a3cd admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.205872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689f8ae9-885b-4e54-a453-213629dfd8d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.206460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689f8ae9-885b-4e54-a453-213629dfd8d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.206734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689f8ae9-885b-4e54-a453-213629dfd8d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.207032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689f8ae9-885b-4e54-a453-213629dfd8d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.207952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689f8ae9-885b-4e54-a453-213629dfd8d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.210377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689f8ae9-885b-4e54-a453-213629dfd8d4 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.220632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1156/2301] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:04.225343 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37989679-2189-4527-9e7b-b0881138450b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.235181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37989679-2189-4527-9e7b-b0881138450b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.240248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4740ed6-55a0-40e0-aaae-5c72a70aedfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.240913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4740ed6-55a0-40e0-aaae-5c72a70aedfb admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.241190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4740ed6-55a0-40e0-aaae-5c72a70aedfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.241462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4740ed6-55a0-40e0-aaae-5c72a70aedfb admin admin] PATH_INFO: `/v3/projects/1cdbc322dbae4343a55e6d5dc1084bf0/users/62697ca506b74f7cba61e63e20e51a02/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.256212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4740ed6-55a0-40e0-aaae-5c72a70aedfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=1cdbc322dbae4343a55e6d5dc1084bf0, user_id=62697ca506b74f7cba61e63e20e51a02, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.258900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4740ed6-55a0-40e0-aaae-5c72a70aedfb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.272467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1146/2302] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:38:04 2022] PUT /identity/v3/projects/1cdbc322dbae4343a55e6d5dc1084bf0/users/62697ca506b74f7cba61e63e20e51a02/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:04.283589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e828e9a1-f3db-4805-bcd8-28e9e111a8ed None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.283974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e828e9a1-f3db-4805-bcd8-28e9e111a8ed None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.284248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e828e9a1-f3db-4805-bcd8-28e9e111a8ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.314967 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e828e9a1-f3db-4805-bcd8-28e9e111a8ed None None] MFA Rules not processed for user `62697ca506b74f7cba61e63e20e51a02`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:04.316187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e828e9a1-f3db-4805-bcd8-28e9e111a8ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:04.379189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1157/2303] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:04 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:04.383810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4740ed6-55a0-40e0-aaae-5c72a70aedfb admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.444488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4740ed6-55a0-40e0-aaae-5c72a70aedfb admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.449350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebfa062-5108-40fb-ab66-3da9b061f4d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.449939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebfa062-5108-40fb-ab66-3da9b061f4d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.450219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebfa062-5108-40fb-ab66-3da9b061f4d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.450495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebfa062-5108-40fb-ab66-3da9b061f4d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.451187 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebfa062-5108-40fb-ab66-3da9b061f4d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.453068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebfa062-5108-40fb-ab66-3da9b061f4d4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.479125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1147/2304] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:38:04 2022] POST /identity/v3/projects => generated 385 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:04.483214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e828e9a1-f3db-4805-bcd8-28e9e111a8ed None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.494569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e828e9a1-f3db-4805-bcd8-28e9e111a8ed None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.550880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c257329f-c7c9-4131-9dae-06c9fb9f5c01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.551489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c257329f-c7c9-4131-9dae-06c9fb9f5c01 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.551782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c257329f-c7c9-4131-9dae-06c9fb9f5c01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.552659 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c257329f-c7c9-4131-9dae-06c9fb9f5c01 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.553296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c257329f-c7c9-4131-9dae-06c9fb9f5c01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.555194 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c257329f-c7c9-4131-9dae-06c9fb9f5c01 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.579575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1158/2305] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:38:04 2022] POST /identity/v3/users => generated 299 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:38:04.580865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebfa062-5108-40fb-ab66-3da9b061f4d4 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.591692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebfa062-5108-40fb-ab66-3da9b061f4d4 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.596290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3337a9a-3cd0-4e92-bfcd-07e28957a3ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.596972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3337a9a-3cd0-4e92-bfcd-07e28957a3ff admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.597254 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3337a9a-3cd0-4e92-bfcd-07e28957a3ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.597527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3337a9a-3cd0-4e92-bfcd-07e28957a3ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.598173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3337a9a-3cd0-4e92-bfcd-07e28957a3ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.600321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3337a9a-3cd0-4e92-bfcd-07e28957a3ff admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.611342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1148/2306] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:04.616136 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c257329f-c7c9-4131-9dae-06c9fb9f5c01 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.626193 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c257329f-c7c9-4131-9dae-06c9fb9f5c01 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.631171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1a397b-7a15-4a09-8b85-5ccbb7ce29b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.631824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1a397b-7a15-4a09-8b85-5ccbb7ce29b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.632106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1a397b-7a15-4a09-8b85-5ccbb7ce29b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.632407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1a397b-7a15-4a09-8b85-5ccbb7ce29b7 admin admin] PATH_INFO: `/v3/projects/723f9a40c05c47dbb540d2af705eabd5/users/5f76d3a937a64bb3bb8c8bd26273a972/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.647405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1a397b-7a15-4a09-8b85-5ccbb7ce29b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=723f9a40c05c47dbb540d2af705eabd5, user_id=5f76d3a937a64bb3bb8c8bd26273a972, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.650090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1a397b-7a15-4a09-8b85-5ccbb7ce29b7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.664576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1159/2307] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:38:04 2022] PUT /identity/v3/projects/723f9a40c05c47dbb540d2af705eabd5/users/5f76d3a937a64bb3bb8c8bd26273a972/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:38:04.668441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3337a9a-3cd0-4e92-bfcd-07e28957a3ff admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.680249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3337a9a-3cd0-4e92-bfcd-07e28957a3ff admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.696546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:04.744183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc327f4-4e5b-4d8b-b1cc-0bbbf459d4ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.744843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc327f4-4e5b-4d8b-b1cc-0bbbf459d4ac admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.745021 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc327f4-4e5b-4d8b-b1cc-0bbbf459d4ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.745292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc327f4-4e5b-4d8b-b1cc-0bbbf459d4ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.745933 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc327f4-4e5b-4d8b-b1cc-0bbbf459d4ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.747521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc327f4-4e5b-4d8b-b1cc-0bbbf459d4ac admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.757240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1149/2308] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:04 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:04.761352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1a397b-7a15-4a09-8b85-5ccbb7ce29b7 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.773593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1a397b-7a15-4a09-8b85-5ccbb7ce29b7 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.778698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8b7170-6116-4cde-a910-99b86074fe17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.779332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8b7170-6116-4cde-a910-99b86074fe17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.779607 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8b7170-6116-4cde-a910-99b86074fe17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.779876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8b7170-6116-4cde-a910-99b86074fe17 admin admin] PATH_INFO: `/v3/projects/723f9a40c05c47dbb540d2af705eabd5/users/5f76d3a937a64bb3bb8c8bd26273a972/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.781806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8b7170-6116-4cde-a910-99b86074fe17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=723f9a40c05c47dbb540d2af705eabd5, user_id=5f76d3a937a64bb3bb8c8bd26273a972, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:04.784452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8b7170-6116-4cde-a910-99b86074fe17 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:04.790533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:04.805903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1160/2309] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:38:04 2022] PUT /identity/v3/projects/723f9a40c05c47dbb540d2af705eabd5/users/5f76d3a937a64bb3bb8c8bd26273a972/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:04.809943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac10bcb-b56c-4491-904f-1d84cbed8a07 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.810229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac10bcb-b56c-4491-904f-1d84cbed8a07 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.810505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac10bcb-b56c-4491-904f-1d84cbed8a07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:04.839495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0ac10bcb-b56c-4491-904f-1d84cbed8a07 None None] MFA Rules not processed for user `5f76d3a937a64bb3bb8c8bd26273a972`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:04.840757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0ac10bcb-b56c-4491-904f-1d84cbed8a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:04.911922 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1150/2310] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:04 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:04.923205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8b7170-6116-4cde-a910-99b86074fe17 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:04.925033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6a8b7170-6116-4cde-a910-99b86074fe17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:04.982745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8b7170-6116-4cde-a910-99b86074fe17 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:04.986412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c681d55-f60f-41d7-b25e-f26d75551278 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:04.987002 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c681d55-f60f-41d7-b25e-f26d75551278 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:04.987281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c681d55-f60f-41d7-b25e-f26d75551278 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:04.987550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c681d55-f60f-41d7-b25e-f26d75551278 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:05.048714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c681d55-f60f-41d7-b25e-f26d75551278 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:05.050521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c681d55-f60f-41d7-b25e-f26d75551278 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:05.060312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1161/2311] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:04 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:06.569538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac10bcb-b56c-4491-904f-1d84cbed8a07 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:06.571023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0ac10bcb-b56c-4491-904f-1d84cbed8a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:06.583166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac10bcb-b56c-4491-904f-1d84cbed8a07 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:06.586739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31464837-e0c3-4511-9975-075b7bf06903 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:06.587329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31464837-e0c3-4511-9975-075b7bf06903 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:06.587610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31464837-e0c3-4511-9975-075b7bf06903 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:06.587879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31464837-e0c3-4511-9975-075b7bf06903 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:06.646477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31464837-e0c3-4511-9975-075b7bf06903 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:06.647991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31464837-e0c3-4511-9975-075b7bf06903 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:06.657321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1151/2312] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:06 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:38:07.263586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c681d55-f60f-41d7-b25e-f26d75551278 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:07.277583 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c681d55-f60f-41d7-b25e-f26d75551278 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:07.281522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a99e96-778b-4ce6-bc61-9246bb66dd3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:07.282205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a99e96-778b-4ce6-bc61-9246bb66dd3b service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:07.282575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a99e96-778b-4ce6-bc61-9246bb66dd3b service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:07.282941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a99e96-778b-4ce6-bc61-9246bb66dd3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:07.293294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a99e96-778b-4ce6-bc61-9246bb66dd3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:07.294991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a99e96-778b-4ce6-bc61-9246bb66dd3b service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:07.306205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1162/2313] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:07 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:09.946339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31464837-e0c3-4511-9975-075b7bf06903 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:09.948229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-31464837-e0c3-4511-9975-075b7bf06903 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:09.993379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:10.021615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31464837-e0c3-4511-9975-075b7bf06903 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:10.026367 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:10.026998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:10.027299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:10.027698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] PATH_INFO: `/v3/users/5669ef04249a4a80842951e2f80cd1af` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:10.029643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] RBAC: Authorizing `identity:delete_user(user_id=5669ef04249a4a80842951e2f80cd1af)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:10.031573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:10.085705 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5669ef04249a4a80842951e2f80cd1af'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.093131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5669ef04249a4a80842951e2f80cd1af'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.093857 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5669ef04249a4a80842951e2f80cd1af'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.094471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5669ef04249a4a80842951e2f80cd1af'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.102225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5669ef04249a4a80842951e2f80cd1af'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.124045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1152/2314] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:09 2022] DELETE /identity/v3/users/5669ef04249a4a80842951e2f80cd1af => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:10.126700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1588d1-353a-4ee9-a431-d1c0bff8dc1b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:10.127182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1588d1-353a-4ee9-a431-d1c0bff8dc1b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:10.127625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1588d1-353a-4ee9-a431-d1c0bff8dc1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:10.152018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-3f1588d1-353a-4ee9-a431-d1c0bff8dc1b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:10.153592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3f1588d1-353a-4ee9-a431-d1c0bff8dc1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:10.243933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1163/2315] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:10.253664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:10.255886 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:10.321175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2f87d4-70e7-4eaf-a77d-95fa7881942b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:10.325180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f745e31-80d7-4744-8d16-aa1e7b4d7e15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:10.325782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f745e31-80d7-4744-8d16-aa1e7b4d7e15 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:10.326060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f745e31-80d7-4744-8d16-aa1e7b4d7e15 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:10.326330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f745e31-80d7-4744-8d16-aa1e7b4d7e15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:10.336729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f745e31-80d7-4744-8d16-aa1e7b4d7e15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:10.338172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f745e31-80d7-4744-8d16-aa1e7b4d7e15 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:10.348054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1153/2316] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:10.685613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1588d1-353a-4ee9-a431-d1c0bff8dc1b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:10.687601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3f1588d1-353a-4ee9-a431-d1c0bff8dc1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:10.711864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1588d1-353a-4ee9-a431-d1c0bff8dc1b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:10.729955 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:10.731592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:10.732079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:10.732387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] PATH_INFO: `/v3/projects/ff33aea8cc614b16bdcd5392504bf09f` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:10.734873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff33aea8cc614b16bdcd5392504bf09f)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:10.738389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:10.783179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:10.794362 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff33aea8cc614b16bdcd5392504bf09f', 'request_id': 'req-dceba32c-8bc0-45a7-9c3c-d31a7a443010'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.800583 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff33aea8cc614b16bdcd5392504bf09f', 'request_id': 'req-dceba32c-8bc0-45a7-9c3c-d31a7a443010'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.809208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff33aea8cc614b16bdcd5392504bf09f', 'request_id': 'req-dceba32c-8bc0-45a7-9c3c-d31a7a443010'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.824542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] The token cache is being invalidate because project ff33aea8cc614b16bdcd5392504bf09f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:10.824835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.825926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.827790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1164/2317] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:10 2022] DELETE /identity/v3/projects/ff33aea8cc614b16bdcd5392504bf09f => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:10.833313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f745e31-80d7-4744-8d16-aa1e7b4d7e15 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:10.843393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f745e31-80d7-4744-8d16-aa1e7b4d7e15 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:10.900723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:10.907225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:10.907972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:10.908252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:10.908601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] PATH_INFO: `/v3/users/27a20294e95e4e10b8a24b23193ec654` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:10.912882 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] RBAC: Authorizing `identity:delete_user(user_id=27a20294e95e4e10b8a24b23193ec654)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:10.915123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:10.977474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27a20294e95e4e10b8a24b23193ec654'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.990186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27a20294e95e4e10b8a24b23193ec654'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.990940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27a20294e95e4e10b8a24b23193ec654'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:10.992088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27a20294e95e4e10b8a24b23193ec654'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.002119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27a20294e95e4e10b8a24b23193ec654'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.024696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1154/2318] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:10 2022] DELETE /identity/v3/users/27a20294e95e4e10b8a24b23193ec654 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:11.399029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:11.399919 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:11.433727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dceba32c-8bc0-45a7-9c3c-d31a7a443010 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:11.452120 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:11.457168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:11.524269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:11.524953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:11.525230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:11.525501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] PATH_INFO: `/v3/projects/263619296b544313949d1e183fbc348e` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:11.526810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] RBAC: Authorizing `identity:delete_project(project_id=263619296b544313949d1e183fbc348e)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:11.528865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:11.564576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09669806-149e-46f7-8c4c-02fcba61912c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:11.576867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:11.578630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:11.582340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:11.583526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] PATH_INFO: `/v3/users/b4744313d73940898b2b79e997d79b7f` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:11.585601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4744313d73940898b2b79e997d79b7f)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:11.587882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '263619296b544313949d1e183fbc348e', 'request_id': 'req-7572fbad-5548-47a3-9371-9e15e9be2932'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.588564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:11.594817 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '263619296b544313949d1e183fbc348e', 'request_id': 'req-7572fbad-5548-47a3-9371-9e15e9be2932'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.609495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '263619296b544313949d1e183fbc348e', 'request_id': 'req-7572fbad-5548-47a3-9371-9e15e9be2932'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.648091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] The token cache is being invalidate because project 263619296b544313949d1e183fbc348e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:11.648769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.649745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7572fbad-5548-47a3-9371-9e15e9be2932 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.652389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1165/2319] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:11 2022] DELETE /identity/v3/projects/263619296b544313949d1e183fbc348e => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:11.660060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86562747-22a4-4437-9822-da38187cbd10 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:11.662477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86562747-22a4-4437-9822-da38187cbd10 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:11.663181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86562747-22a4-4437-9822-da38187cbd10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:11.678404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4744313d73940898b2b79e997d79b7f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.689890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4744313d73940898b2b79e997d79b7f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.691035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4744313d73940898b2b79e997d79b7f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.692930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4744313d73940898b2b79e997d79b7f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.704102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-86562747-22a4-4437-9822-da38187cbd10 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:11.705333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-809c74c2-3cb1-4abd-bef1-5133c46e8dd7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4744313d73940898b2b79e997d79b7f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:11.706068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-86562747-22a4-4437-9822-da38187cbd10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:11.723123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:11.759870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1155/2320] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:11 2022] DELETE /identity/v3/users/b4744313d73940898b2b79e997d79b7f => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:11.764296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00767f1b-0d2d-4216-9b67-d48969436382 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:11.764751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00767f1b-0d2d-4216-9b67-d48969436382 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:11.765191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00767f1b-0d2d-4216-9b67-d48969436382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:11.798522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:11.799725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-00767f1b-0d2d-4216-9b67-d48969436382 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:11.801518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-00767f1b-0d2d-4216-9b67-d48969436382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:11.850268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1166/2321] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:11.855033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bb8595-483f-45c4-a7cb-956d106dddf8 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:11.855033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bb8595-483f-45c4-a7cb-956d106dddf8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:11.855033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bb8595-483f-45c4-a7cb-956d106dddf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:11.891150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-54bb8595-483f-45c4-a7cb-956d106dddf8 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:11.893519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-54bb8595-483f-45c4-a7cb-956d106dddf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:11.894513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1156/2322] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:11.905598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00767f1b-0d2d-4216-9b67-d48969436382 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:11.908316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-00767f1b-0d2d-4216-9b67-d48969436382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:11.912029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1167/2323] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:38:11.916512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bb8595-483f-45c4-a7cb-956d106dddf8 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:11.928130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bb8595-483f-45c4-a7cb-956d106dddf8 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:11.936859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34afcb87-d94f-4a91-b091-a6d54cdf016e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:11.938200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34afcb87-d94f-4a91-b091-a6d54cdf016e admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:11.938490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34afcb87-d94f-4a91-b091-a6d54cdf016e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:11.938764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34afcb87-d94f-4a91-b091-a6d54cdf016e admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:11.939525 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34afcb87-d94f-4a91-b091-a6d54cdf016e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:38:11.939845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34afcb87-d94f-4a91-b091-a6d54cdf016e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:11.941549 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34afcb87-d94f-4a91-b091-a6d54cdf016e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:11.977217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1168/2324] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:38:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:38:11.978650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34afcb87-d94f-4a91-b091-a6d54cdf016e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:11.996719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34afcb87-d94f-4a91-b091-a6d54cdf016e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:12.005194 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59a857a-2aa5-40a2-962a-ccc85dcbffdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:12.005842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a857a-2aa5-40a2-962a-ccc85dcbffdc admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.006186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a857a-2aa5-40a2-962a-ccc85dcbffdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.006467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a857a-2aa5-40a2-962a-ccc85dcbffdc admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.007898 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59a857a-2aa5-40a2-962a-ccc85dcbffdc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:12.010594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59a857a-2aa5-40a2-962a-ccc85dcbffdc admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:12.046689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00767f1b-0d2d-4216-9b67-d48969436382 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:12.063425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd613c0-5a52-486d-b266-e74542159b69 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:12.064557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd613c0-5a52-486d-b266-e74542159b69 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.065032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd613c0-5a52-486d-b266-e74542159b69 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.065696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd613c0-5a52-486d-b266-e74542159b69 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.080436 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1169/2325] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:38:11 2022] POST /identity/v3/projects => generated 369 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:12.086466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59a857a-2aa5-40a2-962a-ccc85dcbffdc admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:12.093960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd613c0-5a52-486d-b266-e74542159b69 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:12.096054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd613c0-5a52-486d-b266-e74542159b69 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:12.125214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1157/2326] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:12.127484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd613c0-5a52-486d-b266-e74542159b69 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:12.131615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4fd613c0-5a52-486d-b266-e74542159b69 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:12.165226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd613c0-5a52-486d-b266-e74542159b69 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:12.293984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:12.294877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.295686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.296653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] PATH_INFO: `/v3/users/6cc96b5420434be5bedded3b80dfd1fa` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.297427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59a857a-2aa5-40a2-962a-ccc85dcbffdc admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:12.303752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6cc96b5420434be5bedded3b80dfd1fa)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:12.306407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d382b04-95f8-4aa0-95fc-62519e58322c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:12.306939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:12.307575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d382b04-95f8-4aa0-95fc-62519e58322c admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.308110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d382b04-95f8-4aa0-95fc-62519e58322c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.308698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d382b04-95f8-4aa0-95fc-62519e58322c admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.310172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d382b04-95f8-4aa0-95fc-62519e58322c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:12.314414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d382b04-95f8-4aa0-95fc-62519e58322c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:12.351921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1170/2327] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:38:12 2022] POST /identity/v3/users => generated 285 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:12.356509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d382b04-95f8-4aa0-95fc-62519e58322c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:12.369051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d382b04-95f8-4aa0-95fc-62519e58322c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:12.381520 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:12.390168 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c85781-960f-46b0-9992-3f6ae3f876fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:12.392509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6cc96b5420434be5bedded3b80dfd1fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:12.392709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c85781-960f-46b0-9992-3f6ae3f876fa admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.393018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c85781-960f-46b0-9992-3f6ae3f876fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.393358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c85781-960f-46b0-9992-3f6ae3f876fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.395918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c85781-960f-46b0-9992-3f6ae3f876fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:12.397910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c85781-960f-46b0-9992-3f6ae3f876fa admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:12.405464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6cc96b5420434be5bedded3b80dfd1fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:12.408502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6cc96b5420434be5bedded3b80dfd1fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:12.409441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6cc96b5420434be5bedded3b80dfd1fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:12.413451 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1171/2328] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:12 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:12.418200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c85781-960f-46b0-9992-3f6ae3f876fa admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:12.420175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b5c85781-960f-46b0-9992-3f6ae3f876fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:12.429868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14644daf-1326-40e6-bcc6-edb3869f99a5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6cc96b5420434be5bedded3b80dfd1fa'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:12.452242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c85781-960f-46b0-9992-3f6ae3f876fa admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:12.466732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1158/2329] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:12 2022] DELETE /identity/v3/users/6cc96b5420434be5bedded3b80dfd1fa => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:12.467382 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a6a9ba-adcd-415d-8463-2f95b3f52367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:12.468709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a6a9ba-adcd-415d-8463-2f95b3f52367 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.468998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a6a9ba-adcd-415d-8463-2f95b3f52367 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.469285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a6a9ba-adcd-415d-8463-2f95b3f52367 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.470411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a6a9ba-adcd-415d-8463-2f95b3f52367 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:12.471509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3ebc78-d9be-41c8-9d4e-d5d124846dbf None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.471763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3ebc78-d9be-41c8-9d4e-d5d124846dbf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.472075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3ebc78-d9be-41c8-9d4e-d5d124846dbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.474032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a6a9ba-adcd-415d-8463-2f95b3f52367 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:12.487499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1172/2330] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:12 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:12.494306 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a6a9ba-adcd-415d-8463-2f95b3f52367 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:12.498810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2d3ebc78-d9be-41c8-9d4e-d5d124846dbf None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:12.500194 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2d3ebc78-d9be-41c8-9d4e-d5d124846dbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:12.509699 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a6a9ba-adcd-415d-8463-2f95b3f52367 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:12.587563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da0d06e-6f97-459a-a7de-d0912990299c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:12.588276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da0d06e-6f97-459a-a7de-d0912990299c admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.588686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da0d06e-6f97-459a-a7de-d0912990299c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.588842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da0d06e-6f97-459a-a7de-d0912990299c admin admin] PATH_INFO: `/v3/projects/894662793ded45fda1dca264e12e2452/users/cad06b4d6ff141808f81786d1836427e/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.609010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da0d06e-6f97-459a-a7de-d0912990299c admin admin] RBAC: Authorizing `identity:create_grant(project_id=894662793ded45fda1dca264e12e2452, user_id=cad06b4d6ff141808f81786d1836427e, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:12.611876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da0d06e-6f97-459a-a7de-d0912990299c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:12.620393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1159/2331] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:12.633264 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3ebc78-d9be-41c8-9d4e-d5d124846dbf None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:12.634066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1173/2332] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:38:12 2022] PUT /identity/v3/projects/894662793ded45fda1dca264e12e2452/users/cad06b4d6ff141808f81786d1836427e/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:12.634782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2d3ebc78-d9be-41c8-9d4e-d5d124846dbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:12.647493 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84475fa9-ecf6-4df5-babc-08fec953ecbe None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.647847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84475fa9-ecf6-4df5-babc-08fec953ecbe None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.648137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84475fa9-ecf6-4df5-babc-08fec953ecbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.656416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:12.690840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-84475fa9-ecf6-4df5-babc-08fec953ecbe None None] MFA Rules not processed for user `cad06b4d6ff141808f81786d1836427e`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:12.693038 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-84475fa9-ecf6-4df5-babc-08fec953ecbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:12.708763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3ebc78-d9be-41c8-9d4e-d5d124846dbf None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:12.788138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1174/2333] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:12 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:38:12.789390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84475fa9-ecf6-4df5-babc-08fec953ecbe None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:12.790681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-84475fa9-ecf6-4df5-babc-08fec953ecbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:12.800398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcce194c-7db9-4f22-9c8d-233d1895e6d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:12.801126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcce194c-7db9-4f22-9c8d-233d1895e6d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.801536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcce194c-7db9-4f22-9c8d-233d1895e6d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.801969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcce194c-7db9-4f22-9c8d-233d1895e6d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.842781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:12.874093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84475fa9-ecf6-4df5-babc-08fec953ecbe None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:12.879095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:12.880555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:12.880840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:12.881186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] PATH_INFO: `/v3/projects/a2fac30e93fc422bb7375ca123b3b307` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:12.882367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2fac30e93fc422bb7375ca123b3b307)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:12.884483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:12.892930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcce194c-7db9-4f22-9c8d-233d1895e6d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:12.900928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcce194c-7db9-4f22-9c8d-233d1895e6d2 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:12.914805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1160/2334] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:38:12.915720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcce194c-7db9-4f22-9c8d-233d1895e6d2 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:12.917168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-dcce194c-7db9-4f22-9c8d-233d1895e6d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:12.953688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2fac30e93fc422bb7375ca123b3b307', 'request_id': 'req-57b028b7-478e-4108-9c88-0106a1307b19'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:12.963860 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2fac30e93fc422bb7375ca123b3b307', 'request_id': 'req-57b028b7-478e-4108-9c88-0106a1307b19'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:12.971004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2fac30e93fc422bb7375ca123b3b307', 'request_id': 'req-57b028b7-478e-4108-9c88-0106a1307b19'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:12.998667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] The token cache is being invalidate because project a2fac30e93fc422bb7375ca123b3b307 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:12.999310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.005913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.008471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1175/2335] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:12 2022] DELETE /identity/v3/projects/a2fac30e93fc422bb7375ca123b3b307 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:13.012379 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:13.014443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:13.026642 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcce194c-7db9-4f22-9c8d-233d1895e6d2 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:13.055331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:13.099847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b028b7-478e-4108-9c88-0106a1307b19 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:13.105434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:13.106068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:13.106342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:13.106634 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] PATH_INFO: `/v3/users/62697ca506b74f7cba61e63e20e51a02` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:13.107746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] RBAC: Authorizing `identity:delete_user(user_id=62697ca506b74f7cba61e63e20e51a02)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:13.109764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:13.115696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b4fb9d-fdfc-4ac4-a706-6e56052d3737 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e07b6890d3004dd78faed48a1ab90086', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:13.116384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b4fb9d-fdfc-4ac4-a706-6e56052d3737 service glance] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:13.116656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b4fb9d-fdfc-4ac4-a706-6e56052d3737 service glance] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:13.116952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b4fb9d-fdfc-4ac4-a706-6e56052d3737 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:13.119464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c9b4fb9d-fdfc-4ac4-a706-6e56052d3737 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:13.177473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '62697ca506b74f7cba61e63e20e51a02'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.186332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '62697ca506b74f7cba61e63e20e51a02'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.187165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '62697ca506b74f7cba61e63e20e51a02'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.187837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '62697ca506b74f7cba61e63e20e51a02'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.188246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b4fb9d-fdfc-4ac4-a706-6e56052d3737 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:13.189951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b4fb9d-fdfc-4ac4-a706-6e56052d3737 service glance] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:13.201091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1161/2336] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:12 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 286 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:13.207392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e9a63805-bbb8-49b0-b6a2-40c285f3b053 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '62697ca506b74f7cba61e63e20e51a02'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.225452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1176/2337] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:13 2022] DELETE /identity/v3/users/62697ca506b74f7cba61e63e20e51a02 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:38:13.288407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b4fb9d-fdfc-4ac4-a706-6e56052d3737 service glance] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:13.292090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c9b4fb9d-fdfc-4ac4-a706-6e56052d3737 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:13.306484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b4fb9d-fdfc-4ac4-a706-6e56052d3737 service glance] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:13.359228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:13.388629 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:13.389373 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:13.389716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:13.390075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] PATH_INFO: `/v3/projects/c67f8f50478749db99129a6b213b4d7f` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:13.391310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c67f8f50478749db99129a6b213b4d7f)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:13.393441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:13.466601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c67f8f50478749db99129a6b213b4d7f', 'request_id': 'req-f1a7e774-f300-4272-9958-0987aad2b5c8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.473022 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c67f8f50478749db99129a6b213b4d7f', 'request_id': 'req-f1a7e774-f300-4272-9958-0987aad2b5c8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.483906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c67f8f50478749db99129a6b213b4d7f', 'request_id': 'req-f1a7e774-f300-4272-9958-0987aad2b5c8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.498082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] The token cache is being invalidate because project c67f8f50478749db99129a6b213b4d7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:13.498457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.499195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.501063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1162/2338] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:13 2022] DELETE /identity/v3/projects/c67f8f50478749db99129a6b213b4d7f => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:13.507329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bd9f56-5933-4421-a600-a2f89779de79 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:13.507847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bd9f56-5933-4421-a600-a2f89779de79 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:13.508003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bd9f56-5933-4421-a600-a2f89779de79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:13.530997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e8bd9f56-5933-4421-a600-a2f89779de79 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:13.532417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e8bd9f56-5933-4421-a600-a2f89779de79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:13.589928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:13.591799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:13.644471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1177/2339] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:13.653640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdfdff1-c109-4166-afdc-f9be72fc87c3 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:13.653925 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdfdff1-c109-4166-afdc-f9be72fc87c3 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:13.654200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdfdff1-c109-4166-afdc-f9be72fc87c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:13.683084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-dfdfdff1-c109-4166-afdc-f9be72fc87c3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:13.684936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-dfdfdff1-c109-4166-afdc-f9be72fc87c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:13.700289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1178/2340] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:13.701339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdfdff1-c109-4166-afdc-f9be72fc87c3 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:13.707327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a7e774-f300-4272-9958-0987aad2b5c8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:13.713256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:13.713989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:13.714281 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:13.714550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] PATH_INFO: `/v3/projects/1cdbc322dbae4343a55e6d5dc1084bf0` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:13.715274 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdfdff1-c109-4166-afdc-f9be72fc87c3 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:13.715825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1cdbc322dbae4343a55e6d5dc1084bf0)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:13.718136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:13.720393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8652b6-7f8a-47db-b516-2c49ff0875d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:13.721112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8652b6-7f8a-47db-b516-2c49ff0875d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:13.721492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8652b6-7f8a-47db-b516-2c49ff0875d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:13.721867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8652b6-7f8a-47db-b516-2c49ff0875d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:13.722716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8652b6-7f8a-47db-b516-2c49ff0875d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:38:13.723142 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8652b6-7f8a-47db-b516-2c49ff0875d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:13.725119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8652b6-7f8a-47db-b516-2c49ff0875d4 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:13.744425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1179/2341] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:38:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:38:13.746944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8652b6-7f8a-47db-b516-2c49ff0875d4 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:13.758429 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8652b6-7f8a-47db-b516-2c49ff0875d4 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:13.764427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a639d78-a10c-43e4-9852-61853e749331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:13.765640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a639d78-a10c-43e4-9852-61853e749331 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:13.765926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a639d78-a10c-43e4-9852-61853e749331 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:13.766265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a639d78-a10c-43e4-9852-61853e749331 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:13.766965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a639d78-a10c-43e4-9852-61853e749331 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:13.771737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a639d78-a10c-43e4-9852-61853e749331 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:13.810437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1cdbc322dbae4343a55e6d5dc1084bf0', 'request_id': 'req-7870cff8-ca79-4e65-a251-9e694cbf6ec6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.817938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1cdbc322dbae4343a55e6d5dc1084bf0', 'request_id': 'req-7870cff8-ca79-4e65-a251-9e694cbf6ec6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.835056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1cdbc322dbae4343a55e6d5dc1084bf0', 'request_id': 'req-7870cff8-ca79-4e65-a251-9e694cbf6ec6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.836875 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:13.842488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:13.850298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1180/2342] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:38:13 2022] POST /identity/v3/projects => generated 371 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:38:13.852594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a639d78-a10c-43e4-9852-61853e749331 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:13.853204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] The token cache is being invalidate because project 1cdbc322dbae4343a55e6d5dc1084bf0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:13.854726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.857730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:13.866419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1163/2343] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:13 2022] DELETE /identity/v3/projects/1cdbc322dbae4343a55e6d5dc1084bf0 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:38:13.868143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:13.869801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:13.939694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a639d78-a10c-43e4-9852-61853e749331 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:13.945172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8332099-1266-4112-9d0a-130bb98d9aa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:13.945793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8332099-1266-4112-9d0a-130bb98d9aa7 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:13.946072 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8332099-1266-4112-9d0a-130bb98d9aa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:13.946347 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8332099-1266-4112-9d0a-130bb98d9aa7 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:13.947305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8332099-1266-4112-9d0a-130bb98d9aa7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:13.949270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8332099-1266-4112-9d0a-130bb98d9aa7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:13.982178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1181/2344] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:38:13 2022] POST /identity/v3/users => generated 286 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:13.983378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7870cff8-ca79-4e65-a251-9e694cbf6ec6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:13.986644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8332099-1266-4112-9d0a-130bb98d9aa7 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:13.989976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:13.990596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:13.990870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:13.991140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] PATH_INFO: `/v3/users/5f76d3a937a64bb3bb8c8bd26273a972` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:13.992368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f76d3a937a64bb3bb8c8bd26273a972)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:13.993387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e8332099-1266-4112-9d0a-130bb98d9aa7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:13.994528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:14.006179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8332099-1266-4112-9d0a-130bb98d9aa7 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:14.011613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa70a471-3924-414e-8c57-e8c7d69217bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:14.012215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa70a471-3924-414e-8c57-e8c7d69217bf admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:14.012539 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa70a471-3924-414e-8c57-e8c7d69217bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:14.012816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa70a471-3924-414e-8c57-e8c7d69217bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:14.014402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa70a471-3924-414e-8c57-e8c7d69217bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:14.015062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa70a471-3924-414e-8c57-e8c7d69217bf admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:14.027095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1182/2345] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:13 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:14.031105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa70a471-3924-414e-8c57-e8c7d69217bf admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:14.041953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa70a471-3924-414e-8c57-e8c7d69217bf admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:14.048001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1171dc49-aa7a-463b-be6e-22addb9fa26e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:14.048736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1171dc49-aa7a-463b-be6e-22addb9fa26e admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:14.049041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1171dc49-aa7a-463b-be6e-22addb9fa26e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:14.049590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1171dc49-aa7a-463b-be6e-22addb9fa26e admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:14.050848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1171dc49-aa7a-463b-be6e-22addb9fa26e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:14.052963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1171dc49-aa7a-463b-be6e-22addb9fa26e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:14.061388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f76d3a937a64bb3bb8c8bd26273a972'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:14.064915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1183/2346] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:14 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:14.069995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1171dc49-aa7a-463b-be6e-22addb9fa26e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:14.073820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f76d3a937a64bb3bb8c8bd26273a972'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:14.075072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f76d3a937a64bb3bb8c8bd26273a972'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:14.076516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f76d3a937a64bb3bb8c8bd26273a972'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:14.081134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1171dc49-aa7a-463b-be6e-22addb9fa26e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:14.088521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900cc3b0-3fcd-4e85-bca9-b8a24697ba08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:14.088861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-634ad7c5-aaaf-4ad2-8725-a817a4fcb985 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f76d3a937a64bb3bb8c8bd26273a972'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:14.089363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900cc3b0-3fcd-4e85-bca9-b8a24697ba08 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:14.089710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900cc3b0-3fcd-4e85-bca9-b8a24697ba08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:14.090068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900cc3b0-3fcd-4e85-bca9-b8a24697ba08 admin admin] PATH_INFO: `/v3/projects/d1d406b57f894053834ce17fa63fa6ac/users/5c72b19a91804de398c476c5e1c404cf/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:14.107515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900cc3b0-3fcd-4e85-bca9-b8a24697ba08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1d406b57f894053834ce17fa63fa6ac, user_id=5c72b19a91804de398c476c5e1c404cf, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:14.111998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900cc3b0-3fcd-4e85-bca9-b8a24697ba08 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:14.120474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1164/2347] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:13 2022] DELETE /identity/v3/users/5f76d3a937a64bb3bb8c8bd26273a972 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:38:14.131182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1184/2348] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:38:14 2022] PUT /identity/v3/projects/d1d406b57f894053834ce17fa63fa6ac/users/5c72b19a91804de398c476c5e1c404cf/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:14.137688 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c98768-35a6-47d7-bbcd-d53f3c7a670f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:14.138304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c98768-35a6-47d7-bbcd-d53f3c7a670f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:14.138939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c98768-35a6-47d7-bbcd-d53f3c7a670f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:14.166228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-39c98768-35a6-47d7-bbcd-d53f3c7a670f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:14.167586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-39c98768-35a6-47d7-bbcd-d53f3c7a670f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:14.260223 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1165/2349] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:14.271325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900cc3b0-3fcd-4e85-bca9-b8a24697ba08 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:14.273418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-900cc3b0-3fcd-4e85-bca9-b8a24697ba08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:14.346489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900cc3b0-3fcd-4e85-bca9-b8a24697ba08 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:14.350367 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43a8ca9-2ae2-4d52-bb6e-e260aea44e52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:14.350980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43a8ca9-2ae2-4d52-bb6e-e260aea44e52 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:14.351258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43a8ca9-2ae2-4d52-bb6e-e260aea44e52 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:14.351532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43a8ca9-2ae2-4d52-bb6e-e260aea44e52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:14.364088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43a8ca9-2ae2-4d52-bb6e-e260aea44e52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:14.365576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43a8ca9-2ae2-4d52-bb6e-e260aea44e52 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:14.375507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1185/2350] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:14.493984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c98768-35a6-47d7-bbcd-d53f3c7a670f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:14.495536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-39c98768-35a6-47d7-bbcd-d53f3c7a670f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:14.509058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c98768-35a6-47d7-bbcd-d53f3c7a670f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:14.514221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:14.515381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:14.515505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:14.515757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] PATH_INFO: `/v3/projects/723f9a40c05c47dbb540d2af705eabd5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:14.517091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=723f9a40c05c47dbb540d2af705eabd5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:14.519048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:14.534675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:14.592228 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '723f9a40c05c47dbb540d2af705eabd5', 'request_id': 'req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:14.598963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '723f9a40c05c47dbb540d2af705eabd5', 'request_id': 'req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:14.606791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '723f9a40c05c47dbb540d2af705eabd5', 'request_id': 'req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:14.636469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] The token cache is being invalidate because project 723f9a40c05c47dbb540d2af705eabd5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:14.637740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:14.641014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:14.651738 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1166/2351] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:14 2022] DELETE /identity/v3/projects/723f9a40c05c47dbb540d2af705eabd5 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:18.967347 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61963a0-b7c7-4653-b661-33d9b65302d5 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:18.967868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61963a0-b7c7-4653-b661-33d9b65302d5 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:18.968042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61963a0-b7c7-4653-b661-33d9b65302d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:19.005562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b61963a0-b7c7-4653-b661-33d9b65302d5 None None] MFA Rules not processed for user `5c72b19a91804de398c476c5e1c404cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:19.006906 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b61963a0-b7c7-4653-b661-33d9b65302d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:19.059156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:19.100161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1186/2352] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:18 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:19.114208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:19.117554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:19.214584 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf85cf0-4a3a-4ee3-853f-618bbdc60df2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:19.220079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a933eec3-6a34-40a0-aa36-2fe10475c1a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:19.220985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a933eec3-6a34-40a0-aa36-2fe10475c1a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:19.221588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a933eec3-6a34-40a0-aa36-2fe10475c1a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:19.221920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a933eec3-6a34-40a0-aa36-2fe10475c1a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:19.223803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a933eec3-6a34-40a0-aa36-2fe10475c1a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:19.226191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a933eec3-6a34-40a0-aa36-2fe10475c1a8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:19.270528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1167/2353] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:38:19 2022] POST /identity/v3/projects => generated 371 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:19.272032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61963a0-b7c7-4653-b661-33d9b65302d5 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:19.288520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61963a0-b7c7-4653-b661-33d9b65302d5 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:19.371252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd155b0-61bd-4f2f-820a-ee717c7dff2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:19.371874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd155b0-61bd-4f2f-820a-ee717c7dff2d admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:19.373874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd155b0-61bd-4f2f-820a-ee717c7dff2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:19.373874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd155b0-61bd-4f2f-820a-ee717c7dff2d admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:19.373874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd155b0-61bd-4f2f-820a-ee717c7dff2d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:19.375728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd155b0-61bd-4f2f-820a-ee717c7dff2d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:19.411870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a933eec3-6a34-40a0-aa36-2fe10475c1a8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:19.413059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1187/2354] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:38:19 2022] POST /identity/v3/users => generated 292 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:38:19.427014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a933eec3-6a34-40a0-aa36-2fe10475c1a8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:19.434659 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99820a22-e132-4afa-bab0-336cdd35b3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:19.435365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99820a22-e132-4afa-bab0-336cdd35b3e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:19.435713 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99820a22-e132-4afa-bab0-336cdd35b3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:19.441148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99820a22-e132-4afa-bab0-336cdd35b3e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:19.442475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99820a22-e132-4afa-bab0-336cdd35b3e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:19.444278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99820a22-e132-4afa-bab0-336cdd35b3e5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:19.467055 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1168/2355] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:19 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:19.474790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd155b0-61bd-4f2f-820a-ee717c7dff2d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:19.492086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd155b0-61bd-4f2f-820a-ee717c7dff2d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:19.505413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f60185d-7700-40f1-9da2-3d34133d846b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:19.506216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f60185d-7700-40f1-9da2-3d34133d846b admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:19.506730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f60185d-7700-40f1-9da2-3d34133d846b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:19.507015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f60185d-7700-40f1-9da2-3d34133d846b admin admin] PATH_INFO: `/v3/projects/020731480e384eb78efd7d5d71df6319/users/a2d5286edc614fc0b69ac0e52879e426/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:19.529507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f60185d-7700-40f1-9da2-3d34133d846b admin admin] RBAC: Authorizing `identity:create_grant(project_id=020731480e384eb78efd7d5d71df6319, user_id=a2d5286edc614fc0b69ac0e52879e426, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:19.532378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f60185d-7700-40f1-9da2-3d34133d846b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:19.551862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1188/2356] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:38:19 2022] PUT /identity/v3/projects/020731480e384eb78efd7d5d71df6319/users/a2d5286edc614fc0b69ac0e52879e426/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:19.555968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99820a22-e132-4afa-bab0-336cdd35b3e5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:19.567026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99820a22-e132-4afa-bab0-336cdd35b3e5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:19.637856 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c71fd3e-a90f-4633-bd32-c439133c1fbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:19.638563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71fd3e-a90f-4633-bd32-c439133c1fbd admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:19.638878 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71fd3e-a90f-4633-bd32-c439133c1fbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:19.639143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71fd3e-a90f-4633-bd32-c439133c1fbd admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:19.639802 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c71fd3e-a90f-4633-bd32-c439133c1fbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:19.641482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c71fd3e-a90f-4633-bd32-c439133c1fbd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:19.654373 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1169/2357] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:19 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:19.658783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f60185d-7700-40f1-9da2-3d34133d846b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:19.670711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f60185d-7700-40f1-9da2-3d34133d846b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:19.675695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57428d1-c400-4308-b57b-7f9fbebb19ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:19.677222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57428d1-c400-4308-b57b-7f9fbebb19ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:19.677492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57428d1-c400-4308-b57b-7f9fbebb19ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:19.677784 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57428d1-c400-4308-b57b-7f9fbebb19ad admin admin] PATH_INFO: `/v3/projects/020731480e384eb78efd7d5d71df6319/users/a2d5286edc614fc0b69ac0e52879e426/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:19.679902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57428d1-c400-4308-b57b-7f9fbebb19ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=020731480e384eb78efd7d5d71df6319, user_id=a2d5286edc614fc0b69ac0e52879e426, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:19.682874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57428d1-c400-4308-b57b-7f9fbebb19ad admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:19.694190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1189/2358] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:38:19 2022] PUT /identity/v3/projects/020731480e384eb78efd7d5d71df6319/users/a2d5286edc614fc0b69ac0e52879e426/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:38:24.968402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398fd530-49e0-41c4-9b4c-fd524e51728f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:24.968830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398fd530-49e0-41c4-9b4c-fd524e51728f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:24.968957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398fd530-49e0-41c4-9b4c-fd524e51728f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:24.994016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:25.007083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-398fd530-49e0-41c4-9b4c-fd524e51728f None None] MFA Rules not processed for user `a2d5286edc614fc0b69ac0e52879e426`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:25.008964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-398fd530-49e0-41c4-9b4c-fd524e51728f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:25.086064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1170/2359] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:24 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:25.100588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57428d1-c400-4308-b57b-7f9fbebb19ad admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:25.102394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f57428d1-c400-4308-b57b-7f9fbebb19ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:25.180489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57428d1-c400-4308-b57b-7f9fbebb19ad admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:25.186093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cf4869-0322-4821-b018-a7b0f47634de service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:25.186696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cf4869-0322-4821-b018-a7b0f47634de service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:25.186976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cf4869-0322-4821-b018-a7b0f47634de service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:25.187249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cf4869-0322-4821-b018-a7b0f47634de service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:25.215934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:25.248862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cf4869-0322-4821-b018-a7b0f47634de service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:25.250465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cf4869-0322-4821-b018-a7b0f47634de service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:25.260645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1190/2360] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:25 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:26.839632 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b84a8d-ed85-4bab-aa75-c7c7442a82cc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:26.840135 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b84a8d-ed85-4bab-aa75-c7c7442a82cc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:26.840242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b84a8d-ed85-4bab-aa75-c7c7442a82cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:26.867323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-42b84a8d-ed85-4bab-aa75-c7c7442a82cc None None] MFA Rules not processed for user `5c72b19a91804de398c476c5e1c404cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:26.868969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-42b84a8d-ed85-4bab-aa75-c7c7442a82cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:26.876811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1171/2361] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:26 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:35.256147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=16 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:35.608435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cf4869-0322-4821-b018-a7b0f47634de service nova] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:35.610570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-65cf4869-0322-4821-b018-a7b0f47634de service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:35.692169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cf4869-0322-4821-b018-a7b0f47634de service nova] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:35.697402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:35.698106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:35.698387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:35.698658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] PATH_INFO: `/v3/users/93f891bbe4dc44cd835eda441f6cf1dd` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:35.699867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] RBAC: Authorizing `identity:delete_user(user_id=93f891bbe4dc44cd835eda441f6cf1dd)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:35.701875 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:35.763422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93f891bbe4dc44cd835eda441f6cf1dd'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:35.774278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93f891bbe4dc44cd835eda441f6cf1dd'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:35.775097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93f891bbe4dc44cd835eda441f6cf1dd'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:35.776076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93f891bbe4dc44cd835eda441f6cf1dd'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:35.795818 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93f891bbe4dc44cd835eda441f6cf1dd'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:35.830716 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1191/2362] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:35 2022] DELETE /identity/v3/users/93f891bbe4dc44cd835eda441f6cf1dd => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:35.835641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd494bf9-c1db-4c34-9745-0c53fe8e87b2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:35.836115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd494bf9-c1db-4c34-9745-0c53fe8e87b2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:35.836238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd494bf9-c1db-4c34-9745-0c53fe8e87b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:35.855490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bd494bf9-c1db-4c34-9745-0c53fe8e87b2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:35.856868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bd494bf9-c1db-4c34-9745-0c53fe8e87b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:35.920845 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1172/2363] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:35.934072 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:35.935857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:36.007631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600f35a3-7e36-4e53-9f08-264eec566387 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:36.011689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec41500f-7371-4cd0-913f-f1b9bbd2a405 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:36.012307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec41500f-7371-4cd0-913f-f1b9bbd2a405 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.012614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec41500f-7371-4cd0-913f-f1b9bbd2a405 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.012884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec41500f-7371-4cd0-913f-f1b9bbd2a405 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.024506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec41500f-7371-4cd0-913f-f1b9bbd2a405 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:36.026117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec41500f-7371-4cd0-913f-f1b9bbd2a405 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:36.038545 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1192/2364] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:36.335526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd494bf9-c1db-4c34-9745-0c53fe8e87b2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:36.337137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bd494bf9-c1db-4c34-9745-0c53fe8e87b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:36.351879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd494bf9-c1db-4c34-9745-0c53fe8e87b2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:36.356735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:36.357358 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.357628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.357907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] PATH_INFO: `/v3/projects/eddc4331819c4814ad83f4d8bcf17d21` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.359071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] RBAC: Authorizing `identity:delete_project(project_id=eddc4331819c4814ad83f4d8bcf17d21)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:36.361148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:36.427309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eddc4331819c4814ad83f4d8bcf17d21', 'request_id': 'req-a92f545b-195e-4c88-a5ad-a09031e4115e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:36.437593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eddc4331819c4814ad83f4d8bcf17d21', 'request_id': 'req-a92f545b-195e-4c88-a5ad-a09031e4115e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:36.464201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eddc4331819c4814ad83f4d8bcf17d21', 'request_id': 'req-a92f545b-195e-4c88-a5ad-a09031e4115e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:36.464201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] The token cache is being invalidate because project eddc4331819c4814ad83f4d8bcf17d21 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:38:36.464201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:36.464864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a92f545b-195e-4c88-a5ad-a09031e4115e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:36.467296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1173/2365] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:38:36 2022] DELETE /identity/v3/projects/eddc4331819c4814ad83f4d8bcf17d21 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:36.472903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfa9bbf-9fde-4e3d-b0a8-d9db7227fe90 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.473393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfa9bbf-9fde-4e3d-b0a8-d9db7227fe90 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.473661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfa9bbf-9fde-4e3d-b0a8-d9db7227fe90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.504681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-6cfa9bbf-9fde-4e3d-b0a8-d9db7227fe90 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:36.506583 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6cfa9bbf-9fde-4e3d-b0a8-d9db7227fe90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:36.528416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:36.585286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cd9954-ce54-41ea-96cd-f2bf1085acec None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.585573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cd9954-ce54-41ea-96cd-f2bf1085acec None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.585846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cd9954-ce54-41ea-96cd-f2bf1085acec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.588734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1193/2366] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:36.610136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f0cd9954-ce54-41ea-96cd-f2bf1085acec None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:36.611425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f0cd9954-ce54-41ea-96cd-f2bf1085acec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:36.620669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1174/2367] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:36.625105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfa9bbf-9fde-4e3d-b0a8-d9db7227fe90 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:36.638065 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfa9bbf-9fde-4e3d-b0a8-d9db7227fe90 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:36.642614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7d8443-565f-43ab-b630-7d62b042ed1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:36.643216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d8443-565f-43ab-b630-7d62b042ed1b admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.643492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d8443-565f-43ab-b630-7d62b042ed1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.644307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d8443-565f-43ab-b630-7d62b042ed1b admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.645383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7d8443-565f-43ab-b630-7d62b042ed1b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:38:36.645687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7d8443-565f-43ab-b630-7d62b042ed1b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:36.647277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7d8443-565f-43ab-b630-7d62b042ed1b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:36.659134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1194/2368] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:38:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:38:36.664328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cd9954-ce54-41ea-96cd-f2bf1085acec None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:36.669904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:36.673745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cd9954-ce54-41ea-96cd-f2bf1085acec None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:36.680050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1688d8-c4d5-4a82-af63-2ca797b568f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:36.680784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1688d8-c4d5-4a82-af63-2ca797b568f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.681363 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1688d8-c4d5-4a82-af63-2ca797b568f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.681870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1688d8-c4d5-4a82-af63-2ca797b568f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.682939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1688d8-c4d5-4a82-af63-2ca797b568f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:36.685003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1688d8-c4d5-4a82-af63-2ca797b568f5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:36.716476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1175/2369] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:38:36 2022] POST /identity/v3/projects => generated 367 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:36.720761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7d8443-565f-43ab-b630-7d62b042ed1b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:36.731366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7d8443-565f-43ab-b630-7d62b042ed1b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:36.796197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a96316-3679-484f-8430-b8c14ec67f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:36.796859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a96316-3679-484f-8430-b8c14ec67f3e admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.797145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a96316-3679-484f-8430-b8c14ec67f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.797411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a96316-3679-484f-8430-b8c14ec67f3e admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.798098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a96316-3679-484f-8430-b8c14ec67f3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:36.800749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a96316-3679-484f-8430-b8c14ec67f3e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:36.826087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1195/2370] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:38:36 2022] POST /identity/v3/users => generated 284 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:38:36.828284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1688d8-c4d5-4a82-af63-2ca797b568f5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:36.838792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1688d8-c4d5-4a82-af63-2ca797b568f5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:36.843532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8d24a9-ee28-4351-8523-662e1ae31405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:36.844091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8d24a9-ee28-4351-8523-662e1ae31405 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.844389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8d24a9-ee28-4351-8523-662e1ae31405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.844675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8d24a9-ee28-4351-8523-662e1ae31405 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.845391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8d24a9-ee28-4351-8523-662e1ae31405 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:36.847457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8d24a9-ee28-4351-8523-662e1ae31405 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:36.859226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1176/2371] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:36.873230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a96316-3679-484f-8430-b8c14ec67f3e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:36.875298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a96316-3679-484f-8430-b8c14ec67f3e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:36.881098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8d1653-3a40-41ec-89ee-c8f7b5fe29b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:36.881704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8d1653-3a40-41ec-89ee-c8f7b5fe29b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.881984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8d1653-3a40-41ec-89ee-c8f7b5fe29b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.882256 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8d1653-3a40-41ec-89ee-c8f7b5fe29b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.882918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8d1653-3a40-41ec-89ee-c8f7b5fe29b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:36.884535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8d1653-3a40-41ec-89ee-c8f7b5fe29b3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:36.895949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1196/2372] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:38:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:38:36.900067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8d24a9-ee28-4351-8523-662e1ae31405 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:36.910504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8d24a9-ee28-4351-8523-662e1ae31405 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:36.916133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ff80d9-f7e3-4b0f-b970-3bf6da94325f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:36.916808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ff80d9-f7e3-4b0f-b970-3bf6da94325f admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.917093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ff80d9-f7e3-4b0f-b970-3bf6da94325f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.917369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ff80d9-f7e3-4b0f-b970-3bf6da94325f admin admin] PATH_INFO: `/v3/projects/3d84eb826e054de3a831769967d5ed91/users/af260a85502047dc960f8ed19df299d6/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.933849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ff80d9-f7e3-4b0f-b970-3bf6da94325f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d84eb826e054de3a831769967d5ed91, user_id=af260a85502047dc960f8ed19df299d6, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:36.937157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ff80d9-f7e3-4b0f-b970-3bf6da94325f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:36.950613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1177/2373] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:38:36 2022] PUT /identity/v3/projects/3d84eb826e054de3a831769967d5ed91/users/af260a85502047dc960f8ed19df299d6/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:38:36.955725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8e3daf-b616-4306-a8d3-53e454d298f8 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:36.956018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8e3daf-b616-4306-a8d3-53e454d298f8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:36.956294 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8e3daf-b616-4306-a8d3-53e454d298f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:36.977116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-2a8e3daf-b616-4306-a8d3-53e454d298f8 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:36.978339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2a8e3daf-b616-4306-a8d3-53e454d298f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:37.052444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1197/2374] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:37.061980 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ff80d9-f7e3-4b0f-b970-3bf6da94325f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:37.063779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d2ff80d9-f7e3-4b0f-b970-3bf6da94325f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:37.124074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ff80d9-f7e3-4b0f-b970-3bf6da94325f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:37.128631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611216d5-11f7-4337-8c8f-0b420c4d1ee5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:37.129652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611216d5-11f7-4337-8c8f-0b420c4d1ee5 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:37.130230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611216d5-11f7-4337-8c8f-0b420c4d1ee5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:37.130874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611216d5-11f7-4337-8c8f-0b420c4d1ee5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:37.140075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611216d5-11f7-4337-8c8f-0b420c4d1ee5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:37.141571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611216d5-11f7-4337-8c8f-0b420c4d1ee5 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:37.151427 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1178/2375] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:38.018179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a25462-279f-4611-8a33-c0a4f01c78a0 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:38.018944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a25462-279f-4611-8a33-c0a4f01c78a0 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:38.019461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a25462-279f-4611-8a33-c0a4f01c78a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:38.051775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-52a25462-279f-4611-8a33-c0a4f01c78a0 None None] MFA Rules not processed for user `af260a85502047dc960f8ed19df299d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:38.053688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-52a25462-279f-4611-8a33-c0a4f01c78a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:38.073694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:38.125859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1198/2376] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:38 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:38.137297 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611216d5-11f7-4337-8c8f-0b420c4d1ee5 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:38.138763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-611216d5-11f7-4337-8c8f-0b420c4d1ee5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:38.202061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611216d5-11f7-4337-8c8f-0b420c4d1ee5 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:38.205880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17566121-926c-48c6-a0fa-aef3ec766762 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:38.206608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17566121-926c-48c6-a0fa-aef3ec766762 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:38.206894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17566121-926c-48c6-a0fa-aef3ec766762 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:38.207172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17566121-926c-48c6-a0fa-aef3ec766762 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:38.216561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17566121-926c-48c6-a0fa-aef3ec766762 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:38.217944 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17566121-926c-48c6-a0fa-aef3ec766762 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:38.226964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1179/2377] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:38 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:39.587352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6b889a-0fe9-49fc-991a-4e874cb0eb15 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:39.587877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6b889a-0fe9-49fc-991a-4e874cb0eb15 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:39.587877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6b889a-0fe9-49fc-991a-4e874cb0eb15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:39.607880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-bc6b889a-0fe9-49fc-991a-4e874cb0eb15 None None] MFA Rules not processed for user `af260a85502047dc960f8ed19df299d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:39.609120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-bc6b889a-0fe9-49fc-991a-4e874cb0eb15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:39.617110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1199/2378] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:39 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:38:41.968110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84581829-6f02-4a79-bfd0-e8728a5a65e1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:41.968648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84581829-6f02-4a79-bfd0-e8728a5a65e1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:41.968829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84581829-6f02-4a79-bfd0-e8728a5a65e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:41.992787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-84581829-6f02-4a79-bfd0-e8728a5a65e1 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:41.994191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-84581829-6f02-4a79-bfd0-e8728a5a65e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:42.071310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:42.108961 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1180/2379] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:38:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:42.152036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6b889a-0fe9-49fc-991a-4e874cb0eb15 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:42.153623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-bc6b889a-0fe9-49fc-991a-4e874cb0eb15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:42.258422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6b889a-0fe9-49fc-991a-4e874cb0eb15 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:42.264020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7445eb16-6367-4b59-b0ab-3baf9a110994 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:42.264667 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7445eb16-6367-4b59-b0ab-3baf9a110994 service swift] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:42.264948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7445eb16-6367-4b59-b0ab-3baf9a110994 service swift] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:42.265221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7445eb16-6367-4b59-b0ab-3baf9a110994 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:42.273489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7445eb16-6367-4b59-b0ab-3baf9a110994 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:42.275164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7445eb16-6367-4b59-b0ab-3baf9a110994 service swift] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:42.283296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1200/2380] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:38:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:38:43.474416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d421c01-f08e-4133-b3ab-613f629ea0ce None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:43.475306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d421c01-f08e-4133-b3ab-613f629ea0ce None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:43.475746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d421c01-f08e-4133-b3ab-613f629ea0ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:43.526278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6d421c01-f08e-4133-b3ab-613f629ea0ce None None] MFA Rules not processed for user `fb54ba1355004e14b8dc43cf3c028e67`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:43.529152 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6d421c01-f08e-4133-b3ab-613f629ea0ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:43.734628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1181/2381] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:43 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:52.280017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=54 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:53.703688 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=26 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:38:59.306201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7445eb16-6367-4b59-b0ab-3baf9a110994 service swift] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:59.307897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7445eb16-6367-4b59-b0ab-3baf9a110994 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:59.323082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7445eb16-6367-4b59-b0ab-3baf9a110994 service swift] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:59.329501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:59.329501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:59.329720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:59.330636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] PATH_INFO: `/v3/users/cad06b4d6ff141808f81786d1836427e` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:59.332008 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cad06b4d6ff141808f81786d1836427e)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:59.334031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:59.400463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cad06b4d6ff141808f81786d1836427e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:59.411013 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cad06b4d6ff141808f81786d1836427e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:59.411913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cad06b4d6ff141808f81786d1836427e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:59.412851 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cad06b4d6ff141808f81786d1836427e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:59.428729 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cad06b4d6ff141808f81786d1836427e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:38:59.445119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1201/2382] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:38:59 2022] DELETE /identity/v3/users/cad06b4d6ff141808f81786d1836427e => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:38:59.451062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4afa651-41c2-4551-9aeb-d52a3cf5aaac None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:59.451588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4afa651-41c2-4551-9aeb-d52a3cf5aaac None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:59.451823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4afa651-41c2-4551-9aeb-d52a3cf5aaac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:59.497512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f4afa651-41c2-4551-9aeb-d52a3cf5aaac None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:59.499755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f4afa651-41c2-4551-9aeb-d52a3cf5aaac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:59.606763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1182/2383] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:38:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:38:59.618115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:38:59.619723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:38:59.694057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a887a588-e8eb-4c04-b65b-11516fbe41f2 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:38:59.699488 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e865725-a487-43dc-8111-df0aa285ae30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:38:59.700171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e865725-a487-43dc-8111-df0aa285ae30 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:59.700582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e865725-a487-43dc-8111-df0aa285ae30 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:59.700928 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e865725-a487-43dc-8111-df0aa285ae30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:59.717655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e865725-a487-43dc-8111-df0aa285ae30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:38:59.719342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e865725-a487-43dc-8111-df0aa285ae30 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:38:59.731543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1202/2384] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:38:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:38:59.955450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596d2811-22f7-4a96-80f3-49ba9e243a14 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:59.956414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596d2811-22f7-4a96-80f3-49ba9e243a14 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:59.956915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596d2811-22f7-4a96-80f3-49ba9e243a14 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:59.959639 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1183/2385] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:38:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:38:59.978029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4d8d13-9183-4445-b775-b38dcacd4817 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:38:59.978029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4d8d13-9183-4445-b775-b38dcacd4817 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:38:59.978029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4d8d13-9183-4445-b775-b38dcacd4817 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:38:59.997406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-1c4d8d13-9183-4445-b775-b38dcacd4817 None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:38:59.998678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1c4d8d13-9183-4445-b775-b38dcacd4817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.068267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596d2811-22f7-4a96-80f3-49ba9e243a14 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.070031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-596d2811-22f7-4a96-80f3-49ba9e243a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.100522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1203/2386] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:38:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:39:00.112182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4d8d13-9183-4445-b775-b38dcacd4817 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.113942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1c4d8d13-9183-4445-b775-b38dcacd4817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.116544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596d2811-22f7-4a96-80f3-49ba9e243a14 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.121110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.121730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.122013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.122315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] PATH_INFO: `/v3/projects/894662793ded45fda1dca264e12e2452` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.123440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] RBAC: Authorizing `identity:delete_project(project_id=894662793ded45fda1dca264e12e2452)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.124161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:00.125556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.175600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4d8d13-9183-4445-b775-b38dcacd4817 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.177200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '894662793ded45fda1dca264e12e2452', 'request_id': 'req-1e52cdda-fe25-4e49-813a-0ffb8da4399b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:00.183773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '894662793ded45fda1dca264e12e2452', 'request_id': 'req-1e52cdda-fe25-4e49-813a-0ffb8da4399b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:00.193954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '894662793ded45fda1dca264e12e2452', 'request_id': 'req-1e52cdda-fe25-4e49-813a-0ffb8da4399b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:00.207662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] The token cache is being invalidate because project 894662793ded45fda1dca264e12e2452 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:39:00.207962 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:00.208666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1e52cdda-fe25-4e49-813a-0ffb8da4399b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:00.210437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1184/2387] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:39:00 2022] DELETE /identity/v3/projects/894662793ded45fda1dca264e12e2452 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:39:00.215354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83cf855-9e1c-47b8-9214-2f029b4e940a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.215354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83cf855-9e1c-47b8-9214-2f029b4e940a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.215598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83cf855-9e1c-47b8-9214-2f029b4e940a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.236383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d83cf855-9e1c-47b8-9214-2f029b4e940a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:39:00.237641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d83cf855-9e1c-47b8-9214-2f029b4e940a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.242096 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d38259-512d-467b-acb9-eb914ed92e09 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e07b6890d3004dd78faed48a1ab90086', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.242875 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d38259-512d-467b-acb9-eb914ed92e09 service glance] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.243736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d38259-512d-467b-acb9-eb914ed92e09 service glance] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.244117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d38259-512d-467b-acb9-eb914ed92e09 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.246184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f4d38259-512d-467b-acb9-eb914ed92e09 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.302352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1185/2388] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:39:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:39:00.310299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142fb579-482d-4d1d-b901-f2179ba93426 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.310595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142fb579-482d-4d1d-b901-f2179ba93426 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.311061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142fb579-482d-4d1d-b901-f2179ba93426 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.329171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d38259-512d-467b-acb9-eb914ed92e09 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.331516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d38259-512d-467b-acb9-eb914ed92e09 service glance] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.332260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-142fb579-482d-4d1d-b901-f2179ba93426 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:39:00.333622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-142fb579-482d-4d1d-b901-f2179ba93426 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.342272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1186/2389] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:39:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:39:00.343444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1204/2390] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:39:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:39:00.346597 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142fb579-482d-4d1d-b901-f2179ba93426 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.356012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142fb579-482d-4d1d-b901-f2179ba93426 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.361154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514d23d3-9471-4cec-abc3-a3925a1d9a73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.361766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514d23d3-9471-4cec-abc3-a3925a1d9a73 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.362046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514d23d3-9471-4cec-abc3-a3925a1d9a73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.362328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514d23d3-9471-4cec-abc3-a3925a1d9a73 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.362976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514d23d3-9471-4cec-abc3-a3925a1d9a73 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:39:00.363284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514d23d3-9471-4cec-abc3-a3925a1d9a73 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.365065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514d23d3-9471-4cec-abc3-a3925a1d9a73 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.377908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1187/2391] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:39:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:39:00.381892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d38259-512d-467b-acb9-eb914ed92e09 service glance] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.392148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d38259-512d-467b-acb9-eb914ed92e09 service glance] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.397626 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b63779-c9df-4553-9580-a31c0953068d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.398472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b63779-c9df-4553-9580-a31c0953068d admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.398780 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b63779-c9df-4553-9580-a31c0953068d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.399056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b63779-c9df-4553-9580-a31c0953068d admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.400075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b63779-c9df-4553-9580-a31c0953068d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.402185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b63779-c9df-4553-9580-a31c0953068d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.419816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09776be9-fb89-40cf-8f74-8e28588e524c None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.420157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09776be9-fb89-40cf-8f74-8e28588e524c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.420499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09776be9-fb89-40cf-8f74-8e28588e524c None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.421835 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1188/2392] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:39:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 26 01:39:00.427633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdf617f-80d4-44b1-92f0-8097c87d135e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.427948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdf617f-80d4-44b1-92f0-8097c87d135e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.428226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdf617f-80d4-44b1-92f0-8097c87d135e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.433675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1205/2393] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:39:00 2022] POST /identity/v3/projects => generated 389 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:39:00.437415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:00.437572 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b63779-c9df-4553-9580-a31c0953068d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.450020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0fdf617f-80d4-44b1-92f0-8097c87d135e None None] MFA Rules not processed for user `0478ebab37fb4f488eef84f7a241e206`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:39:00.451757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0fdf617f-80d4-44b1-92f0-8097c87d135e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.485853 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:00.505578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b63779-c9df-4553-9580-a31c0953068d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.506509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1189/2394] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:39:00 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:39:00.510494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3438e771-afb4-4c71-b125-80884744f476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.511096 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3438e771-afb4-4c71-b125-80884744f476 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.511377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3438e771-afb4-4c71-b125-80884744f476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.511745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3438e771-afb4-4c71-b125-80884744f476 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.512513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3438e771-afb4-4c71-b125-80884744f476 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.514313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3438e771-afb4-4c71-b125-80884744f476 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.519301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdf617f-80d4-44b1-92f0-8097c87d135e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.520681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0fdf617f-80d4-44b1-92f0-8097c87d135e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.532810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdf617f-80d4-44b1-92f0-8097c87d135e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.537873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4162f2-049d-4255-90db-8cabfe67dfd6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.538225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1206/2395] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:39:00 2022] POST /identity/v3/users => generated 295 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:39:00.538478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4162f2-049d-4255-90db-8cabfe67dfd6 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.538786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4162f2-049d-4255-90db-8cabfe67dfd6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.539062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4162f2-049d-4255-90db-8cabfe67dfd6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.542265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3438e771-afb4-4c71-b125-80884744f476 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.546635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4162f2-049d-4255-90db-8cabfe67dfd6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.548083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4162f2-049d-4255-90db-8cabfe67dfd6 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.552870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3438e771-afb4-4c71-b125-80884744f476 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.556643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1190/2396] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:39:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:39:00.557921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7992f790-a950-47d9-b89b-d353f597caca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.558519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7992f790-a950-47d9-b89b-d353f597caca admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.558800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7992f790-a950-47d9-b89b-d353f597caca admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.559073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7992f790-a950-47d9-b89b-d353f597caca admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.559761 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7992f790-a950-47d9-b89b-d353f597caca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.561383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7992f790-a950-47d9-b89b-d353f597caca admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.572437 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1207/2397] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:39:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:39:00.577356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4162f2-049d-4255-90db-8cabfe67dfd6 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.590388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4162f2-049d-4255-90db-8cabfe67dfd6 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.596206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58102b0a-e14e-497b-9ab5-123273d65305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.597174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58102b0a-e14e-497b-9ab5-123273d65305 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.597640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58102b0a-e14e-497b-9ab5-123273d65305 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.597910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58102b0a-e14e-497b-9ab5-123273d65305 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.598554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58102b0a-e14e-497b-9ab5-123273d65305 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.600166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58102b0a-e14e-497b-9ab5-123273d65305 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.610927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1191/2398] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:39:00 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:39:00.616111 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7992f790-a950-47d9-b89b-d353f597caca admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.626479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7992f790-a950-47d9-b89b-d353f597caca admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.631629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a3caf0-5981-4c73-9b49-2d6d8eb1d49c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.632221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3caf0-5981-4c73-9b49-2d6d8eb1d49c admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.632534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3caf0-5981-4c73-9b49-2d6d8eb1d49c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.632818 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3caf0-5981-4c73-9b49-2d6d8eb1d49c admin admin] PATH_INFO: `/v3/projects/c2dac09fc4e64dd88e3c0ff53570f992/users/68c0473fd97e4b758a6601001b7fc633/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.649092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a3caf0-5981-4c73-9b49-2d6d8eb1d49c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2dac09fc4e64dd88e3c0ff53570f992, user_id=68c0473fd97e4b758a6601001b7fc633, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.651873 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a3caf0-5981-4c73-9b49-2d6d8eb1d49c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.653062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0637d1e-e345-45bc-b76b-b43dcdd2fbae None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.653354 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0637d1e-e345-45bc-b76b-b43dcdd2fbae None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.653633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0637d1e-e345-45bc-b76b-b43dcdd2fbae None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.656374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1192/2399] 10.222.0.25 () {54 vars in 921 bytes} [Mon Sep 26 01:39:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 26 01:39:00.664044 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863de123-abdf-43da-939c-f068effa1036 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.664417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863de123-abdf-43da-939c-f068effa1036 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.664814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863de123-abdf-43da-939c-f068effa1036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.671814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1208/2400] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:39:00 2022] PUT /identity/v3/projects/c2dac09fc4e64dd88e3c0ff53570f992/users/68c0473fd97e4b758a6601001b7fc633/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:39:00.683088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ca7922-2a60-48aa-b820-dc2f81bd746a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.683380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ca7922-2a60-48aa-b820-dc2f81bd746a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.683687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ca7922-2a60-48aa-b820-dc2f81bd746a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.706967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-863de123-abdf-43da-939c-f068effa1036 None None] MFA Rules not processed for user `381ad7c239f444da90369dc58779b86c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:39:00.709752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-863de123-abdf-43da-939c-f068effa1036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.720511 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-91ca7922-2a60-48aa-b820-dc2f81bd746a None None] MFA Rules not processed for user `68c0473fd97e4b758a6601001b7fc633`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:39:00.722113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-91ca7922-2a60-48aa-b820-dc2f81bd746a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.812516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1209/2401] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:39:00 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:39:00.824744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1193/2402] 10.222.0.25 () {58 vars in 1018 bytes} [Mon Sep 26 01:39:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:39:00.830694 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ca7922-2a60-48aa-b820-dc2f81bd746a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.832135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-91ca7922-2a60-48aa-b820-dc2f81bd746a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.836298 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863de123-abdf-43da-939c-f068effa1036 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:00.837710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-863de123-abdf-43da-939c-f068effa1036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:00.890441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ca7922-2a60-48aa-b820-dc2f81bd746a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.894247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863de123-abdf-43da-939c-f068effa1036 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:00.894790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18a2b11-17c9-4e0e-8fe6-5c2eaa8bf4f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.895397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18a2b11-17c9-4e0e-8fe6-5c2eaa8bf4f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.895798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18a2b11-17c9-4e0e-8fe6-5c2eaa8bf4f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.896080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18a2b11-17c9-4e0e-8fe6-5c2eaa8bf4f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.898226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426988fc-9d7f-446a-9c6e-2d2f7a0e9447 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:00.898819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426988fc-9d7f-446a-9c6e-2d2f7a0e9447 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:00.899099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426988fc-9d7f-446a-9c6e-2d2f7a0e9447 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:00.899373 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426988fc-9d7f-446a-9c6e-2d2f7a0e9447 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:00.904845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18a2b11-17c9-4e0e-8fe6-5c2eaa8bf4f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.906365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18a2b11-17c9-4e0e-8fe6-5c2eaa8bf4f6 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.910643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426988fc-9d7f-446a-9c6e-2d2f7a0e9447 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:00.912936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426988fc-9d7f-446a-9c6e-2d2f7a0e9447 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:00.916029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1210/2403] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:39:00 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:39:00.924752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1194/2404] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:39:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:39:09.453571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49ff97b-9f6d-495f-9b9f-c895818c1573 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:09.454099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49ff97b-9f6d-495f-9b9f-c895818c1573 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:09.454099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49ff97b-9f6d-495f-9b9f-c895818c1573 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:09.473679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-c49ff97b-9f6d-495f-9b9f-c895818c1573 None None] MFA Rules not processed for user `44d8acef214449a085a2173930bb3ab1`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:39:09.475210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c49ff97b-9f6d-495f-9b9f-c895818c1573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:09.500069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:09.532108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1211/2405] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:39:09 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:39:10.919163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:12.129608 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426988fc-9d7f-446a-9c6e-2d2f7a0e9447 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:12.131800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-426988fc-9d7f-446a-9c6e-2d2f7a0e9447 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:12.328897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49ff97b-9f6d-495f-9b9f-c895818c1573 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:12.332401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c49ff97b-9f6d-495f-9b9f-c895818c1573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:12.343412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426988fc-9d7f-446a-9c6e-2d2f7a0e9447 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:12.353520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49ff97b-9f6d-495f-9b9f-c895818c1573 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:12.354398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086cbf90-8be5-4458-92b3-e348edf87104 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:12.355039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086cbf90-8be5-4458-92b3-e348edf87104 service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:12.355296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086cbf90-8be5-4458-92b3-e348edf87104 service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:12.355587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086cbf90-8be5-4458-92b3-e348edf87104 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:12.363504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-086cbf90-8be5-4458-92b3-e348edf87104 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:12.385740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02db5be-1f67-4f8b-bca6-576ea03481a2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:12.387001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02db5be-1f67-4f8b-bca6-576ea03481a2 service placement] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:12.388790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02db5be-1f67-4f8b-bca6-576ea03481a2 service placement] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:12.389138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02db5be-1f67-4f8b-bca6-576ea03481a2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:12.399509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086cbf90-8be5-4458-92b3-e348edf87104 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:12.401188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086cbf90-8be5-4458-92b3-e348edf87104 service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:12.407072 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02db5be-1f67-4f8b-bca6-576ea03481a2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:12.409602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02db5be-1f67-4f8b-bca6-576ea03481a2 service placement] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:12.423348 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1195/2406] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:39:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:39:12.427595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1212/2407] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:39:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:39:21.676895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086cbf90-8be5-4458-92b3-e348edf87104 service placement] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:21.760775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086cbf90-8be5-4458-92b3-e348edf87104 service placement] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:21.770950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:21.772592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:21.773051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:21.773504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] PATH_INFO: `/v3/users/68c0473fd97e4b758a6601001b7fc633` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:21.775067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] RBAC: Authorizing `identity:delete_user(user_id=68c0473fd97e4b758a6601001b7fc633)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:21.780414 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:21.879948 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '68c0473fd97e4b758a6601001b7fc633'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:21.891188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '68c0473fd97e4b758a6601001b7fc633'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:21.894605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '68c0473fd97e4b758a6601001b7fc633'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:21.896001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '68c0473fd97e4b758a6601001b7fc633'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:21.910144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '68c0473fd97e4b758a6601001b7fc633'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:21.937691 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1196/2408] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:39:21 2022] DELETE /identity/v3/users/68c0473fd97e4b758a6601001b7fc633 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:39:21.942588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917043ee-47d8-4a2f-923a-262f3e750d4c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:21.943103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917043ee-47d8-4a2f-923a-262f3e750d4c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:21.943103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917043ee-47d8-4a2f-923a-262f3e750d4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:21.967743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-917043ee-47d8-4a2f-923a-262f3e750d4c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:39:21.969184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-917043ee-47d8-4a2f-923a-262f3e750d4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:22.088281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1213/2409] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:39:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:39:22.113772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:22.116438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:22.147836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:22.235569 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670761c1-6c52-411b-9cf8-4215cf65278a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:22.239611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5ff9b7-c4f2-4e8f-9688-d526ac19616a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:22.240480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5ff9b7-c4f2-4e8f-9688-d526ac19616a service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:22.240688 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5ff9b7-c4f2-4e8f-9688-d526ac19616a service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:22.240966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5ff9b7-c4f2-4e8f-9688-d526ac19616a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:22.251273 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5ff9b7-c4f2-4e8f-9688-d526ac19616a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:22.252936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5ff9b7-c4f2-4e8f-9688-d526ac19616a service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:22.264222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1197/2410] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:39:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:39:22.925145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917043ee-47d8-4a2f-923a-262f3e750d4c None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:22.927291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-917043ee-47d8-4a2f-923a-262f3e750d4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:22.956715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917043ee-47d8-4a2f-923a-262f3e750d4c None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:22.962345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:22.963320 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:22.963518 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:22.963974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] PATH_INFO: `/v3/projects/c2dac09fc4e64dd88e3c0ff53570f992` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:22.965334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c2dac09fc4e64dd88e3c0ff53570f992)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:22.967571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:23.021670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c2dac09fc4e64dd88e3c0ff53570f992', 'request_id': 'req-1249e6f8-2fee-49eb-a96d-85085014c5b8'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:23.030536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c2dac09fc4e64dd88e3c0ff53570f992', 'request_id': 'req-1249e6f8-2fee-49eb-a96d-85085014c5b8'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:23.042464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c2dac09fc4e64dd88e3c0ff53570f992', 'request_id': 'req-1249e6f8-2fee-49eb-a96d-85085014c5b8'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:23.064681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] The token cache is being invalidate because project c2dac09fc4e64dd88e3c0ff53570f992 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:39:23.064951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7635e0-3948-436e-8cbf-e66f01ad70cb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:23.065399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:23.065491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7635e0-3948-436e-8cbf-e66f01ad70cb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:23.065757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7635e0-3948-436e-8cbf-e66f01ad70cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:23.066068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:39:23.067919 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1214/2411] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:39:22 2022] DELETE /identity/v3/projects/c2dac09fc4e64dd88e3c0ff53570f992 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:39:23.090087 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ef7635e0-3948-436e-8cbf-e66f01ad70cb None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:39:23.091388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ef7635e0-3948-436e-8cbf-e66f01ad70cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:23.159821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1198/2412] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:39:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:39:23.175250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:23.176949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:23.244409 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:23.275158 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1249e6f8-2fee-49eb-a96d-85085014c5b8 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:23.280482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49c9434-1773-41a8-aa1d-4989dd1e6a85 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:23.281674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49c9434-1773-41a8-aa1d-4989dd1e6a85 service swift] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:23.281959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49c9434-1773-41a8-aa1d-4989dd1e6a85 service swift] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:23.282269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49c9434-1773-41a8-aa1d-4989dd1e6a85 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:23.292550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49c9434-1773-41a8-aa1d-4989dd1e6a85 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:23.293989 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49c9434-1773-41a8-aa1d-4989dd1e6a85 service swift] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:23.302933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1215/2413] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:39:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:39:33.151143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=47 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:33.299352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=12 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:34.129309 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b54a2d7-22e0-4db9-87f3-7ff8b05d81ec None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:34.129765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b54a2d7-22e0-4db9-87f3-7ff8b05d81ec None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:34.130051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b54a2d7-22e0-4db9-87f3-7ff8b05d81ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:34.150244 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1b54a2d7-22e0-4db9-87f3-7ff8b05d81ec None None] MFA Rules not processed for user `af260a85502047dc960f8ed19df299d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:39:34.151478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1b54a2d7-22e0-4db9-87f3-7ff8b05d81ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:34.212626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1199/2414] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:39:34 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:39:44.209245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=25 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:46.350417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852510f2-b6ce-4e30-bcac-cae7c1efb6c0 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:46.350877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852510f2-b6ce-4e30-bcac-cae7c1efb6c0 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:46.350987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852510f2-b6ce-4e30-bcac-cae7c1efb6c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:46.370599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-852510f2-b6ce-4e30-bcac-cae7c1efb6c0 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:39:46.371782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-852510f2-b6ce-4e30-bcac-cae7c1efb6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:46.379675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1216/2415] 10.222.0.25 () {58 vars in 967 bytes} [Mon Sep 26 01:39:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:39:46.402867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b54a2d7-22e0-4db9-87f3-7ff8b05d81ec None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:46.415381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b54a2d7-22e0-4db9-87f3-7ff8b05d81ec None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:46.420222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022cb23b-d410-42bf-8643-ce25ab8e6ada service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:46.420840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022cb23b-d410-42bf-8643-ce25ab8e6ada service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:46.421115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022cb23b-d410-42bf-8643-ce25ab8e6ada service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:46.421430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022cb23b-d410-42bf-8643-ce25ab8e6ada service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:46.429653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022cb23b-d410-42bf-8643-ce25ab8e6ada service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:46.432589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022cb23b-d410-42bf-8643-ce25ab8e6ada service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:46.440974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1200/2416] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:39:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:39:49.252150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852510f2-b6ce-4e30-bcac-cae7c1efb6c0 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:39:49.253579 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-852510f2-b6ce-4e30-bcac-cae7c1efb6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:49.308926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852510f2-b6ce-4e30-bcac-cae7c1efb6c0 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:39:49.313216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66236c6-85d3-431e-aace-5b60a4c8c35e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:39:49.313816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66236c6-85d3-431e-aace-5b60a4c8c35e service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:39:49.314094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66236c6-85d3-431e-aace-5b60a4c8c35e service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:39:49.314363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66236c6-85d3-431e-aace-5b60a4c8c35e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:39:49.316079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e66236c6-85d3-431e-aace-5b60a4c8c35e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:39:49.378468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66236c6-85d3-431e-aace-5b60a4c8c35e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:39:49.380006 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66236c6-85d3-431e-aace-5b60a4c8c35e service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:39:49.389857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1217/2417] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:39:49 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:39:56.437757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:39:59.385391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=53 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:06.445131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c9c5ce-dfb1-4b13-8ac9-cb7d976fb2b5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:06.445672 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c9c5ce-dfb1-4b13-8ac9-cb7d976fb2b5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:06.445851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c9c5ce-dfb1-4b13-8ac9-cb7d976fb2b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:06.467232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-05c9c5ce-dfb1-4b13-8ac9-cb7d976fb2b5 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:06.468521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-05c9c5ce-dfb1-4b13-8ac9-cb7d976fb2b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:06.475813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1201/2418] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:40:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:06.488289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66236c6-85d3-431e-aace-5b60a4c8c35e service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:06.499255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66236c6-85d3-431e-aace-5b60a4c8c35e service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:06.505759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe800ea-142a-4b5c-9186-e3b5be1bd39c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:06.506357 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe800ea-142a-4b5c-9186-e3b5be1bd39c service swift] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:06.506636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe800ea-142a-4b5c-9186-e3b5be1bd39c service swift] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:06.506910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe800ea-142a-4b5c-9186-e3b5be1bd39c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:06.516415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe800ea-142a-4b5c-9186-e3b5be1bd39c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:06.518150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe800ea-142a-4b5c-9186-e3b5be1bd39c service swift] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:06.526261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1218/2419] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:40:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:40:16.462371 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:16.522868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=6 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:23.404867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9bbc8d-0518-4a48-a5b4-533681c8036e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:23.405908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9bbc8d-0518-4a48-a5b4-533681c8036e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:23.406161 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9bbc8d-0518-4a48-a5b4-533681c8036e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:23.426697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ef9bbc8d-0518-4a48-a5b4-533681c8036e None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:23.427912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ef9bbc8d-0518-4a48-a5b4-533681c8036e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:23.435613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1202/2420] 10.222.0.25 () {58 vars in 967 bytes} [Mon Sep 26 01:40:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:23.449258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe800ea-142a-4b5c-9186-e3b5be1bd39c service swift] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:23.460131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe800ea-142a-4b5c-9186-e3b5be1bd39c service swift] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:23.465960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e6c022-be06-425b-be9a-92dd950769cc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:23.466840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e6c022-be06-425b-be9a-92dd950769cc service swift] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:23.467108 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e6c022-be06-425b-be9a-92dd950769cc service swift] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:23.467386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e6c022-be06-425b-be9a-92dd950769cc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:23.477032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e6c022-be06-425b-be9a-92dd950769cc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:23.478991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e6c022-be06-425b-be9a-92dd950769cc service swift] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:23.491113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1219/2421] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:40:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:40:25.520353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9bbc8d-0518-4a48-a5b4-533681c8036e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:25.521813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ef9bbc8d-0518-4a48-a5b4-533681c8036e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:25.590719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9bbc8d-0518-4a48-a5b4-533681c8036e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:25.595579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:25.596210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:25.596513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:25.596784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] PATH_INFO: `/v3/users/af260a85502047dc960f8ed19df299d6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:25.597943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] RBAC: Authorizing `identity:delete_user(user_id=af260a85502047dc960f8ed19df299d6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:25.599885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:25.647026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af260a85502047dc960f8ed19df299d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:25.653902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af260a85502047dc960f8ed19df299d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:25.654615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af260a85502047dc960f8ed19df299d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:25.655225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af260a85502047dc960f8ed19df299d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:25.662033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cec7e514-9c9d-41d7-9d81-c5be2cf6274c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af260a85502047dc960f8ed19df299d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:25.679938 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1203/2422] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:40:25 2022] DELETE /identity/v3/users/af260a85502047dc960f8ed19df299d6 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:40:25.856668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e6c022-be06-425b-be9a-92dd950769cc service swift] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:25.871423 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e6c022-be06-425b-be9a-92dd950769cc service swift] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:25.930900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:25.931520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:25.931797 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:25.932067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] PATH_INFO: `/v3/projects/3d84eb826e054de3a831769967d5ed91` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:25.933202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d84eb826e054de3a831769967d5ed91)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:25.935227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:25.980904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d84eb826e054de3a831769967d5ed91', 'request_id': 'req-db145b28-507f-4463-b819-03055a779e25'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:25.986987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d84eb826e054de3a831769967d5ed91', 'request_id': 'req-db145b28-507f-4463-b819-03055a779e25'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:25.992488 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d84eb826e054de3a831769967d5ed91', 'request_id': 'req-db145b28-507f-4463-b819-03055a779e25'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:26.004464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] The token cache is being invalidate because project 3d84eb826e054de3a831769967d5ed91 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:40:26.004759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:26.005374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-db145b28-507f-4463-b819-03055a779e25 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:26.006808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1220/2423] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:40:25 2022] DELETE /identity/v3/projects/3d84eb826e054de3a831769967d5ed91 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:40:26.011702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af913e6-52e0-46b0-857a-95984590a70d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:26.011986 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af913e6-52e0-46b0-857a-95984590a70d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:26.012259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af913e6-52e0-46b0-857a-95984590a70d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:26.033876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4af913e6-52e0-46b0-857a-95984590a70d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:26.035526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4af913e6-52e0-46b0-857a-95984590a70d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:26.099319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1204/2424] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:26.106840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9315b950-75a4-4e4a-baf4-4a8ef1723a03 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:26.107122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9315b950-75a4-4e4a-baf4-4a8ef1723a03 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:26.107387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9315b950-75a4-4e4a-baf4-4a8ef1723a03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:26.128003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-9315b950-75a4-4e4a-baf4-4a8ef1723a03 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:26.129285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9315b950-75a4-4e4a-baf4-4a8ef1723a03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:26.137428 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1221/2425] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:26.141828 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af913e6-52e0-46b0-857a-95984590a70d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:26.151792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af913e6-52e0-46b0-857a-95984590a70d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:26.156596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb47b45-c295-4553-a6e3-f623cef78868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:26.157915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb47b45-c295-4553-a6e3-f623cef78868 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:26.158397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb47b45-c295-4553-a6e3-f623cef78868 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:26.158725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb47b45-c295-4553-a6e3-f623cef78868 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:26.159496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb47b45-c295-4553-a6e3-f623cef78868 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:40:26.159812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb47b45-c295-4553-a6e3-f623cef78868 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:26.161477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb47b45-c295-4553-a6e3-f623cef78868 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:26.170622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:26.174434 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1205/2426] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:40:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:40:26.178710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9315b950-75a4-4e4a-baf4-4a8ef1723a03 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:26.188920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9315b950-75a4-4e4a-baf4-4a8ef1723a03 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:26.194029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e624e6e-8e54-40a9-9fb7-005d2b29e29f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:26.194637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e624e6e-8e54-40a9-9fb7-005d2b29e29f admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:26.194915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e624e6e-8e54-40a9-9fb7-005d2b29e29f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:26.195187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e624e6e-8e54-40a9-9fb7-005d2b29e29f admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:26.195824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e624e6e-8e54-40a9-9fb7-005d2b29e29f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:26.197765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e624e6e-8e54-40a9-9fb7-005d2b29e29f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:26.228241 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1222/2427] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:40:26 2022] POST /identity/v3/projects => generated 365 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:40:26.232465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb47b45-c295-4553-a6e3-f623cef78868 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:26.242996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb47b45-c295-4553-a6e3-f623cef78868 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:26.299811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13acd68-ae35-4882-98eb-fc8bbd6a0bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:26.300337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13acd68-ae35-4882-98eb-fc8bbd6a0bfc admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:26.300668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13acd68-ae35-4882-98eb-fc8bbd6a0bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:26.300994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13acd68-ae35-4882-98eb-fc8bbd6a0bfc admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:26.301656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13acd68-ae35-4882-98eb-fc8bbd6a0bfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:26.303573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13acd68-ae35-4882-98eb-fc8bbd6a0bfc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:26.325833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1206/2428] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:40:26 2022] POST /identity/v3/users => generated 283 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:40:26.330166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e624e6e-8e54-40a9-9fb7-005d2b29e29f admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:26.340369 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e624e6e-8e54-40a9-9fb7-005d2b29e29f admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:26.345491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30afded-90e3-40d8-ba4f-aa14b8d24992 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:26.346092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30afded-90e3-40d8-ba4f-aa14b8d24992 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:26.346368 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30afded-90e3-40d8-ba4f-aa14b8d24992 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:26.346630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30afded-90e3-40d8-ba4f-aa14b8d24992 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:26.347262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30afded-90e3-40d8-ba4f-aa14b8d24992 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:26.348867 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30afded-90e3-40d8-ba4f-aa14b8d24992 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:26.358846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1223/2429] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:40:26 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:40:26.363490 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13acd68-ae35-4882-98eb-fc8bbd6a0bfc admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:26.374796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13acd68-ae35-4882-98eb-fc8bbd6a0bfc admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:26.380638 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d4bc0b-60f6-46c6-b596-6b127017b114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:26.381339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d4bc0b-60f6-46c6-b596-6b127017b114 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:26.381619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d4bc0b-60f6-46c6-b596-6b127017b114 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:26.381894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d4bc0b-60f6-46c6-b596-6b127017b114 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:26.382542 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d4bc0b-60f6-46c6-b596-6b127017b114 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:26.384179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d4bc0b-60f6-46c6-b596-6b127017b114 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:26.395473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1207/2430] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:40:26 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:40:26.399956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30afded-90e3-40d8-ba4f-aa14b8d24992 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:26.410763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30afded-90e3-40d8-ba4f-aa14b8d24992 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:26.421702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46499cb-97a7-4297-a9a2-05ae5602070c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:26.421702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46499cb-97a7-4297-a9a2-05ae5602070c admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:26.421702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46499cb-97a7-4297-a9a2-05ae5602070c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:26.422034 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46499cb-97a7-4297-a9a2-05ae5602070c admin admin] PATH_INFO: `/v3/projects/465acd2125594298b5dfd5abfd56afdb/users/7fda699ae49645b39ac4da042be9d9c4/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:26.448542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46499cb-97a7-4297-a9a2-05ae5602070c admin admin] RBAC: Authorizing `identity:create_grant(project_id=465acd2125594298b5dfd5abfd56afdb, user_id=7fda699ae49645b39ac4da042be9d9c4, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:26.451629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46499cb-97a7-4297-a9a2-05ae5602070c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:26.465575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1224/2431] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:40:26 2022] PUT /identity/v3/projects/465acd2125594298b5dfd5abfd56afdb/users/7fda699ae49645b39ac4da042be9d9c4/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:40:26.532505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e00e10e-45d0-43a4-bb5b-ef54f5e4888b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:26.533110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e00e10e-45d0-43a4-bb5b-ef54f5e4888b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:26.533110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e00e10e-45d0-43a4-bb5b-ef54f5e4888b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:26.561114 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8e00e10e-45d0-43a4-bb5b-ef54f5e4888b None None] MFA Rules not processed for user `7fda699ae49645b39ac4da042be9d9c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:26.562241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8e00e10e-45d0-43a4-bb5b-ef54f5e4888b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:26.626655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1208/2432] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:26 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:26.637648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46499cb-97a7-4297-a9a2-05ae5602070c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:26.639332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d46499cb-97a7-4297-a9a2-05ae5602070c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:26.653697 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:26.698378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46499cb-97a7-4297-a9a2-05ae5602070c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:26.702105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915aed3-59d4-4975-a283-835deb746023 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:26.702700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915aed3-59d4-4975-a283-835deb746023 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:26.702977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915aed3-59d4-4975-a283-835deb746023 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:26.703249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915aed3-59d4-4975-a283-835deb746023 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:26.711632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b915aed3-59d4-4975-a283-835deb746023 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:26.713087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b915aed3-59d4-4975-a283-835deb746023 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:26.722309 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1225/2433] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:40:26 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:40:36.623883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=66,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:36.717698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=20 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:41.481199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e00e10e-45d0-43a4-bb5b-ef54f5e4888b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:41.482656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8e00e10e-45d0-43a4-bb5b-ef54f5e4888b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:41.549757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e00e10e-45d0-43a4-bb5b-ef54f5e4888b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:41.554355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:41.554974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:41.555253 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:41.555533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] PATH_INFO: `/v3/users/44d8acef214449a085a2173930bb3ab1` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:41.556675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44d8acef214449a085a2173930bb3ab1)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:41.558624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:41.603817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44d8acef214449a085a2173930bb3ab1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:41.610922 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44d8acef214449a085a2173930bb3ab1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:41.611611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44d8acef214449a085a2173930bb3ab1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:41.612488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44d8acef214449a085a2173930bb3ab1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:41.619458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9f8fb75e-4540-4850-9cd8-06b32b85ddf5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44d8acef214449a085a2173930bb3ab1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:41.636768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1209/2434] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:40:41 2022] DELETE /identity/v3/users/44d8acef214449a085a2173930bb3ab1 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:40:41.803411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915aed3-59d4-4975-a283-835deb746023 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:41.816036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915aed3-59d4-4975-a283-835deb746023 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:41.872749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:41.873309 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:41.873593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:41.873870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] PATH_INFO: `/v3/projects/db1abf10c3e24836a161f3035d7e58ca` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:41.874996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] RBAC: Authorizing `identity:delete_project(project_id=db1abf10c3e24836a161f3035d7e58ca)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:41.876959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:41.923598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db1abf10c3e24836a161f3035d7e58ca', 'request_id': 'req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:41.930722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db1abf10c3e24836a161f3035d7e58ca', 'request_id': 'req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:41.936377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db1abf10c3e24836a161f3035d7e58ca', 'request_id': 'req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:41.952723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] The token cache is being invalidate because project db1abf10c3e24836a161f3035d7e58ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:40:41.953012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:41.953671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ea88735a-7c6e-46db-bfa2-e06b9f91c96d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:41.955215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1226/2435] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:40:41 2022] DELETE /identity/v3/projects/db1abf10c3e24836a161f3035d7e58ca => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:40:41.960229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb0216c-7b62-4bdb-b83b-4e163f101674 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:41.960599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb0216c-7b62-4bdb-b83b-4e163f101674 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:41.960879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb0216c-7b62-4bdb-b83b-4e163f101674 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:41.981829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-cbb0216c-7b62-4bdb-b83b-4e163f101674 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:41.983366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cbb0216c-7b62-4bdb-b83b-4e163f101674 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:42.079349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1210/2436] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:42.087052 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74913291-0722-4b21-9f7a-2c37b594aad6 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:42.087366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74913291-0722-4b21-9f7a-2c37b594aad6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:42.087681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74913291-0722-4b21-9f7a-2c37b594aad6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:42.113810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-74913291-0722-4b21-9f7a-2c37b594aad6 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:42.115633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-74913291-0722-4b21-9f7a-2c37b594aad6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:42.128221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1227/2437] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:42.128983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb0216c-7b62-4bdb-b83b-4e163f101674 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:42.141030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb0216c-7b62-4bdb-b83b-4e163f101674 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:42.151246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f56e25-e4a6-4e63-905b-88b53449ed7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:42.151894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f56e25-e4a6-4e63-905b-88b53449ed7a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:42.152170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f56e25-e4a6-4e63-905b-88b53449ed7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:42.152773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f56e25-e4a6-4e63-905b-88b53449ed7a admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:42.153456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f56e25-e4a6-4e63-905b-88b53449ed7a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:40:42.154320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f56e25-e4a6-4e63-905b-88b53449ed7a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:42.156575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f56e25-e4a6-4e63-905b-88b53449ed7a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:42.172405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1211/2438] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:40:42.175267 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74913291-0722-4b21-9f7a-2c37b594aad6 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:42.196010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74913291-0722-4b21-9f7a-2c37b594aad6 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:42.196010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4b6be0-76f3-440d-90b8-6f4e30ba333f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:42.196010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b6be0-76f3-440d-90b8-6f4e30ba333f admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:42.196010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b6be0-76f3-440d-90b8-6f4e30ba333f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:42.196657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b6be0-76f3-440d-90b8-6f4e30ba333f admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:42.196657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4b6be0-76f3-440d-90b8-6f4e30ba333f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:42.196657 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4b6be0-76f3-440d-90b8-6f4e30ba333f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:42.233032 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1228/2439] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:40:42 2022] POST /identity/v3/projects => generated 397 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:40:42.236974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f56e25-e4a6-4e63-905b-88b53449ed7a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:42.247225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f56e25-e4a6-4e63-905b-88b53449ed7a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:42.252332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:42.323332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2e191a-5fd3-45cc-b219-afaad58a6603 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:42.324008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2e191a-5fd3-45cc-b219-afaad58a6603 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:42.325595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2e191a-5fd3-45cc-b219-afaad58a6603 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:42.325883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2e191a-5fd3-45cc-b219-afaad58a6603 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:42.326695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2e191a-5fd3-45cc-b219-afaad58a6603 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:42.328699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2e191a-5fd3-45cc-b219-afaad58a6603 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:42.351855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1212/2440] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:40:42 2022] POST /identity/v3/users => generated 299 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:40:42.355820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4b6be0-76f3-440d-90b8-6f4e30ba333f admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:42.366200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4b6be0-76f3-440d-90b8-6f4e30ba333f admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:42.371505 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23e733f-2246-4c3d-9166-ea826eaaab20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:42.372151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e733f-2246-4c3d-9166-ea826eaaab20 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:42.372497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e733f-2246-4c3d-9166-ea826eaaab20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:42.372774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e733f-2246-4c3d-9166-ea826eaaab20 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:42.373606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23e733f-2246-4c3d-9166-ea826eaaab20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:42.375512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23e733f-2246-4c3d-9166-ea826eaaab20 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:42.388277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1229/2441] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:40:42.391969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2e191a-5fd3-45cc-b219-afaad58a6603 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:42.405731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2e191a-5fd3-45cc-b219-afaad58a6603 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:42.411466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e642c02-911e-48ae-ba7b-87f5bbe19bab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:42.412231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e642c02-911e-48ae-ba7b-87f5bbe19bab admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:42.412545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e642c02-911e-48ae-ba7b-87f5bbe19bab admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:42.412821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e642c02-911e-48ae-ba7b-87f5bbe19bab admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:42.413470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e642c02-911e-48ae-ba7b-87f5bbe19bab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:42.415073 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e642c02-911e-48ae-ba7b-87f5bbe19bab admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:42.426681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1213/2442] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:40:42.430962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23e733f-2246-4c3d-9166-ea826eaaab20 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:42.441701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23e733f-2246-4c3d-9166-ea826eaaab20 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:42.446599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c9375e-eeea-43be-b982-30d1f5e39dba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:42.447244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c9375e-eeea-43be-b982-30d1f5e39dba admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:42.447523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c9375e-eeea-43be-b982-30d1f5e39dba admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:42.447798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c9375e-eeea-43be-b982-30d1f5e39dba admin admin] PATH_INFO: `/v3/projects/9a1697f1642a405aa411ae44acfb70af/users/cd025fd7e0cd4cdf9cb9acca6ba453a1/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:42.466156 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c9375e-eeea-43be-b982-30d1f5e39dba admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a1697f1642a405aa411ae44acfb70af, user_id=cd025fd7e0cd4cdf9cb9acca6ba453a1, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:42.468875 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c9375e-eeea-43be-b982-30d1f5e39dba admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:42.484587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1230/2443] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:40:42 2022] PUT /identity/v3/projects/9a1697f1642a405aa411ae44acfb70af/users/cd025fd7e0cd4cdf9cb9acca6ba453a1/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:40:42.553626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385b995c-36af-442e-aa43-2c17a1542a2a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:42.554091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385b995c-36af-442e-aa43-2c17a1542a2a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:42.554200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385b995c-36af-442e-aa43-2c17a1542a2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:42.585981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-385b995c-36af-442e-aa43-2c17a1542a2a None None] MFA Rules not processed for user `cd025fd7e0cd4cdf9cb9acca6ba453a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:42.587234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-385b995c-36af-442e-aa43-2c17a1542a2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:42.651186 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1214/2444] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:42 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:42.663273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c9375e-eeea-43be-b982-30d1f5e39dba admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:42.664990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-47c9375e-eeea-43be-b982-30d1f5e39dba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:42.696624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:42.723857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c9375e-eeea-43be-b982-30d1f5e39dba admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:42.727677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b29b8f-abc8-41a5-9543-c16cf942987b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:42.728276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b29b8f-abc8-41a5-9543-c16cf942987b service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:42.728589 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b29b8f-abc8-41a5-9543-c16cf942987b service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:42.728872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b29b8f-abc8-41a5-9543-c16cf942987b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:42.738224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b29b8f-abc8-41a5-9543-c16cf942987b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:42.739680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b29b8f-abc8-41a5-9543-c16cf942987b service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:42.748597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1231/2445] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:40:42 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:40:49.914221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385b995c-36af-442e-aa43-2c17a1542a2a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:49.981933 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385b995c-36af-442e-aa43-2c17a1542a2a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:49.986436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:49.987170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:49.987460 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:49.987734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] PATH_INFO: `/v3/users/cd025fd7e0cd4cdf9cb9acca6ba453a1` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:49.989455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd025fd7e0cd4cdf9cb9acca6ba453a1)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:49.991415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:50.009530 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:50.047379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd025fd7e0cd4cdf9cb9acca6ba453a1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:50.058406 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd025fd7e0cd4cdf9cb9acca6ba453a1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:50.059350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd025fd7e0cd4cdf9cb9acca6ba453a1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:50.060115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd025fd7e0cd4cdf9cb9acca6ba453a1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:50.073573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd025fd7e0cd4cdf9cb9acca6ba453a1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:50.107032 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1215/2446] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:40:49 2022] DELETE /identity/v3/users/cd025fd7e0cd4cdf9cb9acca6ba453a1 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:40:50.111715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bc674a-c2b2-49d2-b6f8-7e4c7cd2bdd2 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:50.112255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bc674a-c2b2-49d2-b6f8-7e4c7cd2bdd2 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:50.112455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bc674a-c2b2-49d2-b6f8-7e4c7cd2bdd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:50.131763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-36bc674a-c2b2-49d2-b6f8-7e4c7cd2bdd2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:50.133132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-36bc674a-c2b2-49d2-b6f8-7e4c7cd2bdd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:50.195297 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1232/2447] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:50.205212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:50.206688 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:50.268888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ef2698-afa5-4ea2-af54-408a0e347cf3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:50.277234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed35e09-e22b-4dd5-9072-3cf1e06ed2db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:50.277855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed35e09-e22b-4dd5-9072-3cf1e06ed2db service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:50.278129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed35e09-e22b-4dd5-9072-3cf1e06ed2db service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:50.278398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed35e09-e22b-4dd5-9072-3cf1e06ed2db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:50.289533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed35e09-e22b-4dd5-9072-3cf1e06ed2db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:50.291018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed35e09-e22b-4dd5-9072-3cf1e06ed2db service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:50.300910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1216/2448] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:40:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:40:50.739671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bc674a-c2b2-49d2-b6f8-7e4c7cd2bdd2 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:50.742003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-36bc674a-c2b2-49d2-b6f8-7e4c7cd2bdd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:50.756620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bc674a-c2b2-49d2-b6f8-7e4c7cd2bdd2 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:50.772203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:50.773009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:50.773299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:50.773566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] PATH_INFO: `/v3/projects/9a1697f1642a405aa411ae44acfb70af` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:50.775331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a1697f1642a405aa411ae44acfb70af)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:50.778526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:50.794499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57bad56-7c27-47ea-9584-e4c4e61039a3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:50.794499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57bad56-7c27-47ea-9584-e4c4e61039a3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:50.794499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57bad56-7c27-47ea-9584-e4c4e61039a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:50.814397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e57bad56-7c27-47ea-9584-e4c4e61039a3 None None] MFA Rules not processed for user `bc62747b8aab40d9bff9d853b9b73199`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:50.815751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e57bad56-7c27-47ea-9584-e4c4e61039a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:50.858554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a1697f1642a405aa411ae44acfb70af', 'request_id': 'req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:50.866328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a1697f1642a405aa411ae44acfb70af', 'request_id': 'req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:50.873291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a1697f1642a405aa411ae44acfb70af', 'request_id': 'req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:50.878949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1217/2449] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:50 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:40:50.890828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] The token cache is being invalidate because project 9a1697f1642a405aa411ae44acfb70af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:40:50.890828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:50.891515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9e49c8d5-62f0-4db6-aa3f-724dbe6ac14f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:40:50.896490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1233/2450] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:40:50 2022] DELETE /identity/v3/projects/9a1697f1642a405aa411ae44acfb70af => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:40:50.899852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527410b6-5d7f-463a-819a-b54c1b298237 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:50.900145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527410b6-5d7f-463a-819a-b54c1b298237 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:50.900459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527410b6-5d7f-463a-819a-b54c1b298237 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:50.923093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-527410b6-5d7f-463a-819a-b54c1b298237 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:50.924317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-527410b6-5d7f-463a-819a-b54c1b298237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:50.973133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:50.996857 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1218/2451] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:40:51.004668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e61578-1fac-4405-b5c6-ec0e671e281e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:51.005029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e61578-1fac-4405-b5c6-ec0e671e281e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:51.005358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e61578-1fac-4405-b5c6-ec0e671e281e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:51.026745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-54e61578-1fac-4405-b5c6-ec0e671e281e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:51.027944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-54e61578-1fac-4405-b5c6-ec0e671e281e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:51.040568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1234/2452] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:51.042794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527410b6-5d7f-463a-819a-b54c1b298237 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:51.054380 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527410b6-5d7f-463a-819a-b54c1b298237 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:51.060081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232963e8-8eb4-47b3-976f-bd0c4ad109e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:51.061567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232963e8-8eb4-47b3-976f-bd0c4ad109e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:51.061897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232963e8-8eb4-47b3-976f-bd0c4ad109e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:51.062178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232963e8-8eb4-47b3-976f-bd0c4ad109e7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:51.062839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232963e8-8eb4-47b3-976f-bd0c4ad109e7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:40:51.063144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232963e8-8eb4-47b3-976f-bd0c4ad109e7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:51.064799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232963e8-8eb4-47b3-976f-bd0c4ad109e7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:51.077739 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1219/2453] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:40:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:40:51.082653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e61578-1fac-4405-b5c6-ec0e671e281e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:51.093653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e61578-1fac-4405-b5c6-ec0e671e281e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:51.101916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0bc098-b851-4719-8922-716fbacfb92e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:51.103015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0bc098-b851-4719-8922-716fbacfb92e admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:51.103494 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0bc098-b851-4719-8922-716fbacfb92e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:51.103907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0bc098-b851-4719-8922-716fbacfb92e admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:51.104704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0bc098-b851-4719-8922-716fbacfb92e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:51.107636 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0bc098-b851-4719-8922-716fbacfb92e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:51.151426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1235/2454] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:40:51 2022] POST /identity/v3/projects => generated 377 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:40:51.157774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232963e8-8eb4-47b3-976f-bd0c4ad109e7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:51.169397 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232963e8-8eb4-47b3-976f-bd0c4ad109e7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:51.236697 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be13503a-727a-46da-bcd0-680d9782acb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:51.237420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be13503a-727a-46da-bcd0-680d9782acb8 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:51.237720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be13503a-727a-46da-bcd0-680d9782acb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:51.238014 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be13503a-727a-46da-bcd0-680d9782acb8 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:51.239225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be13503a-727a-46da-bcd0-680d9782acb8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:51.241177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be13503a-727a-46da-bcd0-680d9782acb8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:51.270064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1220/2455] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:40:51 2022] POST /identity/v3/users => generated 289 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:40:51.274101 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0bc098-b851-4719-8922-716fbacfb92e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:51.284513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0bc098-b851-4719-8922-716fbacfb92e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:51.289865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddfbebf-9a71-4f03-9027-4c20c29515b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:51.290470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddfbebf-9a71-4f03-9027-4c20c29515b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:51.290747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddfbebf-9a71-4f03-9027-4c20c29515b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:51.291167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddfbebf-9a71-4f03-9027-4c20c29515b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:51.291798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddfbebf-9a71-4f03-9027-4c20c29515b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:51.293440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddfbebf-9a71-4f03-9027-4c20c29515b1 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:51.307245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1236/2456] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:40:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:40:51.309861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be13503a-727a-46da-bcd0-680d9782acb8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:51.321740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be13503a-727a-46da-bcd0-680d9782acb8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:51.326892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0117ad79-7091-4cb5-af1e-5909ab5dfc03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:51.327570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0117ad79-7091-4cb5-af1e-5909ab5dfc03 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:51.327995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0117ad79-7091-4cb5-af1e-5909ab5dfc03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:51.328289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0117ad79-7091-4cb5-af1e-5909ab5dfc03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:51.328997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0117ad79-7091-4cb5-af1e-5909ab5dfc03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:51.330613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0117ad79-7091-4cb5-af1e-5909ab5dfc03 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:51.342226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1221/2457] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:40:51 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:40:51.346460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddfbebf-9a71-4f03-9027-4c20c29515b1 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:51.357053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddfbebf-9a71-4f03-9027-4c20c29515b1 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:51.362119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da667a9-91a3-4539-bcb5-df14ebb07e7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:51.362785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da667a9-91a3-4539-bcb5-df14ebb07e7d admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:51.363103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da667a9-91a3-4539-bcb5-df14ebb07e7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:51.363386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da667a9-91a3-4539-bcb5-df14ebb07e7d admin admin] PATH_INFO: `/v3/projects/786ed33f027946f581147e672b947df0/users/a47ec612bce54bc8b2062b03d36d10d4/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:51.375490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:51.380513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da667a9-91a3-4539-bcb5-df14ebb07e7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=786ed33f027946f581147e672b947df0, user_id=a47ec612bce54bc8b2062b03d36d10d4, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:51.383635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da667a9-91a3-4539-bcb5-df14ebb07e7d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:51.399380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1237/2458] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:40:51 2022] PUT /identity/v3/projects/786ed33f027946f581147e672b947df0/users/a47ec612bce54bc8b2062b03d36d10d4/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:40:51.404585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e272c6dd-e955-4408-8768-5f6a7dbb1dae None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:51.404903 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e272c6dd-e955-4408-8768-5f6a7dbb1dae None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:51.405224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e272c6dd-e955-4408-8768-5f6a7dbb1dae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:51.424646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e272c6dd-e955-4408-8768-5f6a7dbb1dae None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:51.425936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e272c6dd-e955-4408-8768-5f6a7dbb1dae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:51.491701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1222/2459] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:51.502731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da667a9-91a3-4539-bcb5-df14ebb07e7d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:51.506998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3da667a9-91a3-4539-bcb5-df14ebb07e7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:51.571881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da667a9-91a3-4539-bcb5-df14ebb07e7d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:51.576415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8d082c-1d84-4f12-a354-7bc62ea1daa8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:51.577292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8d082c-1d84-4f12-a354-7bc62ea1daa8 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:51.577610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8d082c-1d84-4f12-a354-7bc62ea1daa8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:51.577976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8d082c-1d84-4f12-a354-7bc62ea1daa8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:51.587492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8d082c-1d84-4f12-a354-7bc62ea1daa8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:51.588884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8d082c-1d84-4f12-a354-7bc62ea1daa8 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:51.602656 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1238/2460] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:40:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:40:56.232538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a43340-a670-43e4-8e43-4bfa19dbc763 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:56.233061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a43340-a670-43e4-8e43-4bfa19dbc763 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:56.233786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a43340-a670-43e4-8e43-4bfa19dbc763 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:56.262616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c2a43340-a670-43e4-8e43-4bfa19dbc763 None None] MFA Rules not processed for user `a47ec612bce54bc8b2062b03d36d10d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:40:56.263847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c2a43340-a670-43e4-8e43-4bfa19dbc763 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:40:56.304722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:40:56.333028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1223/2461] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:40:56 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:40:56.337644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8d082c-1d84-4f12-a354-7bc62ea1daa8 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:56.347308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8d082c-1d84-4f12-a354-7bc62ea1daa8 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:56.351936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af36e773-05e0-45a5-a914-c5dfdae9933a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:56.352560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af36e773-05e0-45a5-a914-c5dfdae9933a admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:56.352844 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af36e773-05e0-45a5-a914-c5dfdae9933a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:56.353118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af36e773-05e0-45a5-a914-c5dfdae9933a admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:56.353744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af36e773-05e0-45a5-a914-c5dfdae9933a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:56.355660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af36e773-05e0-45a5-a914-c5dfdae9933a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:56.385031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1239/2462] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:40:56 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:40:56.389340 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a43340-a670-43e4-8e43-4bfa19dbc763 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:56.401759 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a43340-a670-43e4-8e43-4bfa19dbc763 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:56.479261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c9dc93-a9f7-48d2-b145-b975b209162f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:56.479970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c9dc93-a9f7-48d2-b145-b975b209162f admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:56.480324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c9dc93-a9f7-48d2-b145-b975b209162f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:56.480698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c9dc93-a9f7-48d2-b145-b975b209162f admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:56.481413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c9dc93-a9f7-48d2-b145-b975b209162f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:56.483450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c9dc93-a9f7-48d2-b145-b975b209162f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:56.507063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1224/2463] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:40:56 2022] POST /identity/v3/users => generated 295 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:40:56.510708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af36e773-05e0-45a5-a914-c5dfdae9933a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:56.524880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af36e773-05e0-45a5-a914-c5dfdae9933a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:56.530820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62898d98-e104-459c-b3ff-68ed9a7efa06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:56.531568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62898d98-e104-459c-b3ff-68ed9a7efa06 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:56.531856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62898d98-e104-459c-b3ff-68ed9a7efa06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:56.532134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62898d98-e104-459c-b3ff-68ed9a7efa06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:56.532855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62898d98-e104-459c-b3ff-68ed9a7efa06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:56.535310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62898d98-e104-459c-b3ff-68ed9a7efa06 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:56.546601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1240/2464] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:40:56 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:40:56.551478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c9dc93-a9f7-48d2-b145-b975b209162f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:56.569155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c9dc93-a9f7-48d2-b145-b975b209162f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:56.572275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53751f94-2645-48ae-b30b-7768244fae51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:56.573093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53751f94-2645-48ae-b30b-7768244fae51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:56.573444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53751f94-2645-48ae-b30b-7768244fae51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:56.573789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53751f94-2645-48ae-b30b-7768244fae51 admin admin] PATH_INFO: `/v3/projects/fd28bfe672a742128e64c0b0badce537/users/848cc8f602ad47189c95c80b307f6700/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:56.597840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53751f94-2645-48ae-b30b-7768244fae51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd28bfe672a742128e64c0b0badce537, user_id=848cc8f602ad47189c95c80b307f6700, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:56.600644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53751f94-2645-48ae-b30b-7768244fae51 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:56.641861 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1225/2465] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:40:56 2022] PUT /identity/v3/projects/fd28bfe672a742128e64c0b0badce537/users/848cc8f602ad47189c95c80b307f6700/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:40:56.646147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62898d98-e104-459c-b3ff-68ed9a7efa06 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:56.656640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62898d98-e104-459c-b3ff-68ed9a7efa06 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:56.732456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f9f301-aa8f-4d86-b0ed-c879274d9e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:56.733186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f9f301-aa8f-4d86-b0ed-c879274d9e8c admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:56.733515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f9f301-aa8f-4d86-b0ed-c879274d9e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:56.733794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f9f301-aa8f-4d86-b0ed-c879274d9e8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:56.734449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f9f301-aa8f-4d86-b0ed-c879274d9e8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:56.736178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f9f301-aa8f-4d86-b0ed-c879274d9e8c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:56.752626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53751f94-2645-48ae-b30b-7768244fae51 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:40:56.753046 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1241/2466] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:40:56 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:40:56.765087 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53751f94-2645-48ae-b30b-7768244fae51 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:40:56.771652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c191269-131b-4dc6-9208-b320f131bc59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:40:56.772529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c191269-131b-4dc6-9208-b320f131bc59 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:40:56.772923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c191269-131b-4dc6-9208-b320f131bc59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:40:56.773292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c191269-131b-4dc6-9208-b320f131bc59 admin admin] PATH_INFO: `/v3/projects/fd28bfe672a742128e64c0b0badce537/users/848cc8f602ad47189c95c80b307f6700/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:40:56.775372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c191269-131b-4dc6-9208-b320f131bc59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd28bfe672a742128e64c0b0badce537, user_id=848cc8f602ad47189c95c80b307f6700, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:40:56.778382 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c191269-131b-4dc6-9208-b320f131bc59 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:40:56.792012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1226/2467] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:40:56 2022] PUT /identity/v3/projects/fd28bfe672a742128e64c0b0badce537/users/848cc8f602ad47189c95c80b307f6700/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:41:02.404886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7a98d1-a086-4ffd-843b-c4d872baf72f None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:02.405399 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7a98d1-a086-4ffd-843b-c4d872baf72f None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:02.405559 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7a98d1-a086-4ffd-843b-c4d872baf72f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:02.438986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-6b7a98d1-a086-4ffd-843b-c4d872baf72f None None] MFA Rules not processed for user `848cc8f602ad47189c95c80b307f6700`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:41:02.444300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6b7a98d1-a086-4ffd-843b-c4d872baf72f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:02.495896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:02.525210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1242/2468] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:41:02 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:41:02.540787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c191269-131b-4dc6-9208-b320f131bc59 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:02.546964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5c191269-131b-4dc6-9208-b320f131bc59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:02.629742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c191269-131b-4dc6-9208-b320f131bc59 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:02.635130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ee3b29-be93-4bc0-8488-a43bf5190fed service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:02.635731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ee3b29-be93-4bc0-8488-a43bf5190fed service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:02.636013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ee3b29-be93-4bc0-8488-a43bf5190fed service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:02.636290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ee3b29-be93-4bc0-8488-a43bf5190fed service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:02.664058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:02.706991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ee3b29-be93-4bc0-8488-a43bf5190fed service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:02.708841 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ee3b29-be93-4bc0-8488-a43bf5190fed service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:02.718667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1227/2469] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:02 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:02.743663 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7a98d1-a086-4ffd-843b-c4d872baf72f None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:02.808903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7a98d1-a086-4ffd-843b-c4d872baf72f None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:02.812812 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae4dbdf-5f80-45bd-b7f8-0fc44fe08b84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:02.813542 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae4dbdf-5f80-45bd-b7f8-0fc44fe08b84 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:02.814239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae4dbdf-5f80-45bd-b7f8-0fc44fe08b84 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:02.814239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae4dbdf-5f80-45bd-b7f8-0fc44fe08b84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:02.816142 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4ae4dbdf-5f80-45bd-b7f8-0fc44fe08b84 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:02.831945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae4dbdf-5f80-45bd-b7f8-0fc44fe08b84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:02.834473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae4dbdf-5f80-45bd-b7f8-0fc44fe08b84 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:02.848165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1243/2470] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:03.042963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db11c9b9-de38-4384-82dd-f8fd476f844f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:03.043229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db11c9b9-de38-4384-82dd-f8fd476f844f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:03.043510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db11c9b9-de38-4384-82dd-f8fd476f844f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:03.070976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-db11c9b9-de38-4384-82dd-f8fd476f844f None None] MFA Rules not processed for user `fb54ba1355004e14b8dc43cf3c028e67`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:41:03.072603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-db11c9b9-de38-4384-82dd-f8fd476f844f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:03.146441 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1228/2471] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:41:03 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:41:04.995335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae4dbdf-5f80-45bd-b7f8-0fc44fe08b84 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:05.006148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae4dbdf-5f80-45bd-b7f8-0fc44fe08b84 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:05.010612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee40fc51-2a50-463b-a93b-88d5d64f63f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:05.011506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee40fc51-2a50-463b-a93b-88d5d64f63f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:05.012014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee40fc51-2a50-463b-a93b-88d5d64f63f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:05.012672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee40fc51-2a50-463b-a93b-88d5d64f63f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:05.014981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ee40fc51-2a50-463b-a93b-88d5d64f63f5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:05.038748 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee40fc51-2a50-463b-a93b-88d5d64f63f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:05.040621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee40fc51-2a50-463b-a93b-88d5d64f63f5 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:05.055776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1244/2472] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:05.582837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db11c9b9-de38-4384-82dd-f8fd476f844f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:05.584923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-db11c9b9-de38-4384-82dd-f8fd476f844f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:05.688144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db11c9b9-de38-4384-82dd-f8fd476f844f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:05.704930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:05.705605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:05.705922 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:05.706369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] PATH_INFO: `/v3/users/7fda699ae49645b39ac4da042be9d9c4` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:05.710942 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] RBAC: Authorizing `identity:delete_user(user_id=7fda699ae49645b39ac4da042be9d9c4)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:05.713428 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:05.781114 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7fda699ae49645b39ac4da042be9d9c4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:05.796705 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7fda699ae49645b39ac4da042be9d9c4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:05.797710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7fda699ae49645b39ac4da042be9d9c4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:05.801791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7fda699ae49645b39ac4da042be9d9c4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:05.816621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7fda699ae49645b39ac4da042be9d9c4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:05.835800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:05.858825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1229/2473] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:41:05 2022] DELETE /identity/v3/users/7fda699ae49645b39ac4da042be9d9c4 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:41:05.865386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf356b-be29-4ec8-b4fd-db9520e1f359 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:05.866687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf356b-be29-4ec8-b4fd-db9520e1f359 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:05.867652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf356b-be29-4ec8-b4fd-db9520e1f359 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:05.905500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-4ebf356b-be29-4ec8-b4fd-db9520e1f359 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:41:05.906759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4ebf356b-be29-4ec8-b4fd-db9520e1f359 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:05.981614 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1245/2474] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:41:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:41:05.994729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:05.996390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:06.085667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29da2407-1f11-454f-9790-f31161b7e6bd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:06.095305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5768725-4eb3-4bce-99b6-4c62a8e5ae68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:06.095950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5768725-4eb3-4bce-99b6-4c62a8e5ae68 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:06.096235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5768725-4eb3-4bce-99b6-4c62a8e5ae68 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:06.096763 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5768725-4eb3-4bce-99b6-4c62a8e5ae68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:06.114649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5768725-4eb3-4bce-99b6-4c62a8e5ae68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:06.116121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5768725-4eb3-4bce-99b6-4c62a8e5ae68 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:06.128498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1230/2475] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:06.512212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebf356b-be29-4ec8-b4fd-db9520e1f359 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:06.512212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4ebf356b-be29-4ec8-b4fd-db9520e1f359 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:06.539559 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebf356b-be29-4ec8-b4fd-db9520e1f359 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:06.564495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:06.565178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:06.570484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:06.571504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] PATH_INFO: `/v3/projects/465acd2125594298b5dfd5abfd56afdb` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:06.573175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=465acd2125594298b5dfd5abfd56afdb)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:06.575291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:06.669257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '465acd2125594298b5dfd5abfd56afdb', 'request_id': 'req-71751785-d7db-4438-ad6e-dab5694db2e5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:06.680949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '465acd2125594298b5dfd5abfd56afdb', 'request_id': 'req-71751785-d7db-4438-ad6e-dab5694db2e5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:06.691779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '465acd2125594298b5dfd5abfd56afdb', 'request_id': 'req-71751785-d7db-4438-ad6e-dab5694db2e5'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:06.695759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:06.719586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] The token cache is being invalidate because project 465acd2125594298b5dfd5abfd56afdb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:41:06.722572 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:06.724439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-71751785-d7db-4438-ad6e-dab5694db2e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:06.728405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1246/2476] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:41:06 2022] DELETE /identity/v3/projects/465acd2125594298b5dfd5abfd56afdb => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:41:06.732226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7394c1b2-7f92-484f-b930-e3b3b80f3c2a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:06.732839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7394c1b2-7f92-484f-b930-e3b3b80f3c2a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:06.732839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7394c1b2-7f92-484f-b930-e3b3b80f3c2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:06.766113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7394c1b2-7f92-484f-b930-e3b3b80f3c2a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:41:06.770167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7394c1b2-7f92-484f-b930-e3b3b80f3c2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:06.897275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1231/2477] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:41:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:41:06.899043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1d6506-1dc3-4f2a-9ac6-a981b35f7c61 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:06.899327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1d6506-1dc3-4f2a-9ac6-a981b35f7c61 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:06.899608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1d6506-1dc3-4f2a-9ac6-a981b35f7c61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:06.950885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-1b1d6506-1dc3-4f2a-9ac6-a981b35f7c61 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:41:06.952515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1b1d6506-1dc3-4f2a-9ac6-a981b35f7c61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:06.964413 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1247/2478] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:41:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:41:06.965806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7394c1b2-7f92-484f-b930-e3b3b80f3c2a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:06.981333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7394c1b2-7f92-484f-b930-e3b3b80f3c2a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:06.989024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb18d0f7-b5bd-47b4-8fea-70c785891047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:06.989651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb18d0f7-b5bd-47b4-8fea-70c785891047 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:06.990317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb18d0f7-b5bd-47b4-8fea-70c785891047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:06.990605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb18d0f7-b5bd-47b4-8fea-70c785891047 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:06.991464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb18d0f7-b5bd-47b4-8fea-70c785891047 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:41:06.991779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb18d0f7-b5bd-47b4-8fea-70c785891047 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:06.993503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb18d0f7-b5bd-47b4-8fea-70c785891047 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:07.008449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1232/2479] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:41:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:41:07.016277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1d6506-1dc3-4f2a-9ac6-a981b35f7c61 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:07.027193 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1d6506-1dc3-4f2a-9ac6-a981b35f7c61 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:07.033724 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5789685-677c-4d38-a984-9e35397a0c94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:07.035496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5789685-677c-4d38-a984-9e35397a0c94 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:07.035776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5789685-677c-4d38-a984-9e35397a0c94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:07.040290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5789685-677c-4d38-a984-9e35397a0c94 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:07.041654 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5789685-677c-4d38-a984-9e35397a0c94 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:07.043805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5789685-677c-4d38-a984-9e35397a0c94 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:07.087932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1248/2480] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:41:07 2022] POST /identity/v3/projects => generated 371 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:41:07.095341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb18d0f7-b5bd-47b4-8fea-70c785891047 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:07.112359 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb18d0f7-b5bd-47b4-8fea-70c785891047 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:07.214981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d93e0f-0c5a-4218-b8df-7c987e8928f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:07.215604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d93e0f-0c5a-4218-b8df-7c987e8928f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:07.215887 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d93e0f-0c5a-4218-b8df-7c987e8928f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:07.216703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d93e0f-0c5a-4218-b8df-7c987e8928f7 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:07.217710 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d93e0f-0c5a-4218-b8df-7c987e8928f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:07.219712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d93e0f-0c5a-4218-b8df-7c987e8928f7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:07.256416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1233/2481] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:41:07 2022] POST /identity/v3/users => generated 286 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:41:07.260396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5789685-677c-4d38-a984-9e35397a0c94 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:07.273615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5789685-677c-4d38-a984-9e35397a0c94 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:07.278711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900f20c3-6764-48e6-b5bd-c8a8ad755e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:07.279340 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900f20c3-6764-48e6-b5bd-c8a8ad755e57 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:07.279752 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900f20c3-6764-48e6-b5bd-c8a8ad755e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:07.280100 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900f20c3-6764-48e6-b5bd-c8a8ad755e57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:07.280872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900f20c3-6764-48e6-b5bd-c8a8ad755e57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:07.282953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900f20c3-6764-48e6-b5bd-c8a8ad755e57 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:07.295547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1249/2482] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:41:07 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:41:07.300301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d93e0f-0c5a-4218-b8df-7c987e8928f7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:07.312438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d93e0f-0c5a-4218-b8df-7c987e8928f7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:07.319534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ae8d94-a2c4-4720-9ca1-5846a476dc15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:07.320545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ae8d94-a2c4-4720-9ca1-5846a476dc15 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:07.320998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ae8d94-a2c4-4720-9ca1-5846a476dc15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:07.321694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ae8d94-a2c4-4720-9ca1-5846a476dc15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:07.322526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ae8d94-a2c4-4720-9ca1-5846a476dc15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:07.324437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ae8d94-a2c4-4720-9ca1-5846a476dc15 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:07.335265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:07.350245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900f20c3-6764-48e6-b5bd-c8a8ad755e57 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:07.356508 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1234/2483] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:41:07 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:41:07.363061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900f20c3-6764-48e6-b5bd-c8a8ad755e57 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:07.370948 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9bae12-576c-42cb-8a1d-3ae7fe894c38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:07.371783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9bae12-576c-42cb-8a1d-3ae7fe894c38 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:07.372167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9bae12-576c-42cb-8a1d-3ae7fe894c38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:07.372595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9bae12-576c-42cb-8a1d-3ae7fe894c38 admin admin] PATH_INFO: `/v3/projects/decaa8aea076462684cb489129e50f74/users/81e4316e762c460fa923b06b78c2ec87/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:07.396139 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9bae12-576c-42cb-8a1d-3ae7fe894c38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=decaa8aea076462684cb489129e50f74, user_id=81e4316e762c460fa923b06b78c2ec87, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:07.401090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9bae12-576c-42cb-8a1d-3ae7fe894c38 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:07.420391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1250/2484] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:41:07 2022] PUT /identity/v3/projects/decaa8aea076462684cb489129e50f74/users/81e4316e762c460fa923b06b78c2ec87/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:41:07.434030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bdc5e9-c365-4a7f-9b1e-e7e5dcb88fa6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:07.434338 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bdc5e9-c365-4a7f-9b1e-e7e5dcb88fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:07.434595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bdc5e9-c365-4a7f-9b1e-e7e5dcb88fa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:07.475863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b8bdc5e9-c365-4a7f-9b1e-e7e5dcb88fa6 None None] MFA Rules not processed for user `81e4316e762c460fa923b06b78c2ec87`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:41:07.477387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b8bdc5e9-c365-4a7f-9b1e-e7e5dcb88fa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:07.571945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1235/2485] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:41:07 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:41:07.590338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9bae12-576c-42cb-8a1d-3ae7fe894c38 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:07.592091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ce9bae12-576c-42cb-8a1d-3ae7fe894c38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:07.663305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9bae12-576c-42cb-8a1d-3ae7fe894c38 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:07.668762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc22f8d-b55a-430f-87a1-0a8e57bad9a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:07.669510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc22f8d-b55a-430f-87a1-0a8e57bad9a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:07.670018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc22f8d-b55a-430f-87a1-0a8e57bad9a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:07.670374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc22f8d-b55a-430f-87a1-0a8e57bad9a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:07.685878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc22f8d-b55a-430f-87a1-0a8e57bad9a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:07.688095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc22f8d-b55a-430f-87a1-0a8e57bad9a8 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:07.706488 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1251/2486] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:07 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:17.566445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=32 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:17.696110 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:DELETE=1,keystone:SELECT=55,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:18.034812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bdc5e9-c365-4a7f-9b1e-e7e5dcb88fa6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:18.100534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bdc5e9-c365-4a7f-9b1e-e7e5dcb88fa6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:18.105002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:18.105625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:18.105904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:18.106180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] PATH_INFO: `/v3/users/81e4316e762c460fa923b06b78c2ec87` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:18.107300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] RBAC: Authorizing `identity:delete_user(user_id=81e4316e762c460fa923b06b78c2ec87)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:18.109276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:18.156024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '81e4316e762c460fa923b06b78c2ec87'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:18.165232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '81e4316e762c460fa923b06b78c2ec87'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:18.165862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '81e4316e762c460fa923b06b78c2ec87'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:18.166566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '81e4316e762c460fa923b06b78c2ec87'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:18.174033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '81e4316e762c460fa923b06b78c2ec87'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:18.192394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1236/2487] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:41:18 2022] DELETE /identity/v3/users/81e4316e762c460fa923b06b78c2ec87 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:41:18.197101 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a306b538-f53a-425c-b6de-9b47939bef61 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:18.197389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a306b538-f53a-425c-b6de-9b47939bef61 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:18.197663 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a306b538-f53a-425c-b6de-9b47939bef61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:18.216087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-a306b538-f53a-425c-b6de-9b47939bef61 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:41:18.217327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a306b538-f53a-425c-b6de-9b47939bef61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:18.276986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1252/2488] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:41:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:41:18.287082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:18.288760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:18.345571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80503813-29a2-4307-9432-04c844bdc59c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:18.349868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7539c49-fe86-46df-9106-244654b0c97f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:18.350463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7539c49-fe86-46df-9106-244654b0c97f service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:18.350745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7539c49-fe86-46df-9106-244654b0c97f service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:18.351019 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7539c49-fe86-46df-9106-244654b0c97f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:18.360119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7539c49-fe86-46df-9106-244654b0c97f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:18.361589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7539c49-fe86-46df-9106-244654b0c97f service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:18.371770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1237/2489] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:41:18.644074 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a306b538-f53a-425c-b6de-9b47939bef61 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:18.645519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a306b538-f53a-425c-b6de-9b47939bef61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:18.659299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a306b538-f53a-425c-b6de-9b47939bef61 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:18.663880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:18.665140 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:18.665587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:18.665876 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] PATH_INFO: `/v3/projects/decaa8aea076462684cb489129e50f74` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:18.667112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] RBAC: Authorizing `identity:delete_project(project_id=decaa8aea076462684cb489129e50f74)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:18.669078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:18.716977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'decaa8aea076462684cb489129e50f74', 'request_id': 'req-08088335-9efa-4b2e-ba70-57f00e3ba27d'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:18.723522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'decaa8aea076462684cb489129e50f74', 'request_id': 'req-08088335-9efa-4b2e-ba70-57f00e3ba27d'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:18.729069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'decaa8aea076462684cb489129e50f74', 'request_id': 'req-08088335-9efa-4b2e-ba70-57f00e3ba27d'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:18.742755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] The token cache is being invalidate because project decaa8aea076462684cb489129e50f74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:41:18.743040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:18.743677 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:41:18.745168 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1253/2490] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:41:18 2022] DELETE /identity/v3/projects/decaa8aea076462684cb489129e50f74 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:41:21.405621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6796ae-a84e-4e51-af11-b88028c6302e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:21.406176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6796ae-a84e-4e51-af11-b88028c6302e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:21.406176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6796ae-a84e-4e51-af11-b88028c6302e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:21.426599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3e6796ae-a84e-4e51-af11-b88028c6302e None None] MFA Rules not processed for user `a47ec612bce54bc8b2062b03d36d10d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:41:21.428050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3e6796ae-a84e-4e51-af11-b88028c6302e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:21.446570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:21.498533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1238/2491] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:41:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:41:24.410981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:24.412379 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:24.509702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08088335-9efa-4b2e-ba70-57f00e3ba27d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:24.515334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd90d21-15d8-4afc-afe8-d5424640b7fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:24.515944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd90d21-15d8-4afc-afe8-d5424640b7fc service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:24.516226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd90d21-15d8-4afc-afe8-d5424640b7fc service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:24.516540 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd90d21-15d8-4afc-afe8-d5424640b7fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:24.518308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-bbd90d21-15d8-4afc-afe8-d5424640b7fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:24.540819 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:24.582187 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd90d21-15d8-4afc-afe8-d5424640b7fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:24.583785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd90d21-15d8-4afc-afe8-d5424640b7fc service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:24.593152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1254/2492] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:24 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:31.491845 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=15 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:31.967486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363de681-8fec-447b-84c1-cefb138c06f4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:31.967833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363de681-8fec-447b-84c1-cefb138c06f4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:31.968112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363de681-8fec-447b-84c1-cefb138c06f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:31.989669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-363de681-8fec-447b-84c1-cefb138c06f4 None None] MFA Rules not processed for user `06043e862a394764a58003f205cbb484`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:41:31.990868 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-363de681-8fec-447b-84c1-cefb138c06f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:31.997627 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1239/2493] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:41:31 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:41:32.309933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd90d21-15d8-4afc-afe8-d5424640b7fc service nova] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:32.311395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-bbd90d21-15d8-4afc-afe8-d5424640b7fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:32.368510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd90d21-15d8-4afc-afe8-d5424640b7fc service nova] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:32.371935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acb3a52-4d13-4b31-b33e-a3e6442cbbaa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:32.372567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acb3a52-4d13-4b31-b33e-a3e6442cbbaa service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:32.372843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acb3a52-4d13-4b31-b33e-a3e6442cbbaa service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:32.373112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acb3a52-4d13-4b31-b33e-a3e6442cbbaa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:32.374849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8acb3a52-4d13-4b31-b33e-a3e6442cbbaa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:32.384713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acb3a52-4d13-4b31-b33e-a3e6442cbbaa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:32.386119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acb3a52-4d13-4b31-b33e-a3e6442cbbaa service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:32.394815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1255/2494] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:32 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:32.717153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363de681-8fec-447b-84c1-cefb138c06f4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:32.718618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-363de681-8fec-447b-84c1-cefb138c06f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:32.791781 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363de681-8fec-447b-84c1-cefb138c06f4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:32.796766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68685966-21fa-4f9a-bdba-ca7fc9f93cf7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:32.797365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68685966-21fa-4f9a-bdba-ca7fc9f93cf7 service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:32.797648 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68685966-21fa-4f9a-bdba-ca7fc9f93cf7 service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:32.797919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68685966-21fa-4f9a-bdba-ca7fc9f93cf7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:32.799700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-68685966-21fa-4f9a-bdba-ca7fc9f93cf7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:32.810997 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68685966-21fa-4f9a-bdba-ca7fc9f93cf7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:32.812641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68685966-21fa-4f9a-bdba-ca7fc9f93cf7 service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:32.823109 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1240/2495] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:32 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:36.501104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acb3a52-4d13-4b31-b33e-a3e6442cbbaa service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:36.510930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acb3a52-4d13-4b31-b33e-a3e6442cbbaa service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:36.515994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04542400-41ca-462f-bf16-090441ed3470 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:36.516740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04542400-41ca-462f-bf16-090441ed3470 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:36.517120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04542400-41ca-462f-bf16-090441ed3470 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:36.517526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04542400-41ca-462f-bf16-090441ed3470 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:36.520005 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-04542400-41ca-462f-bf16-090441ed3470 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:36.530623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04542400-41ca-462f-bf16-090441ed3470 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:36.532039 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04542400-41ca-462f-bf16-090441ed3470 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:36.540991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1256/2496] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:36 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:40.578415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68685966-21fa-4f9a-bdba-ca7fc9f93cf7 service placement] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:40.587806 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68685966-21fa-4f9a-bdba-ca7fc9f93cf7 service placement] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:40.591377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2894821-ef53-4194-84c5-21f260155036 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:40.591960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2894821-ef53-4194-84c5-21f260155036 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:40.592238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2894821-ef53-4194-84c5-21f260155036 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:40.592540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2894821-ef53-4194-84c5-21f260155036 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:40.594234 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a2894821-ef53-4194-84c5-21f260155036 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:40.656588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2894821-ef53-4194-84c5-21f260155036 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:40.658018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2894821-ef53-4194-84c5-21f260155036 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:40.667766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1241/2497] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:40 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:42.733377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04542400-41ca-462f-bf16-090441ed3470 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:42.734777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-04542400-41ca-462f-bf16-090441ed3470 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:42.751462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04542400-41ca-462f-bf16-090441ed3470 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:42.758242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336cf53-853d-4754-a162-19af628a6094 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:42.758929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336cf53-853d-4754-a162-19af628a6094 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:42.759143 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336cf53-853d-4754-a162-19af628a6094 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:42.759414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336cf53-853d-4754-a162-19af628a6094 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:42.761313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3336cf53-853d-4754-a162-19af628a6094 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:42.774049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336cf53-853d-4754-a162-19af628a6094 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:42.775641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336cf53-853d-4754-a162-19af628a6094 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:42.786376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1257/2498] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:44.219595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2894821-ef53-4194-84c5-21f260155036 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:44.232007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2894821-ef53-4194-84c5-21f260155036 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:44.238489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d70cf4c-a047-41d3-862e-28a2affff072 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:44.239089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d70cf4c-a047-41d3-862e-28a2affff072 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:44.239361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d70cf4c-a047-41d3-862e-28a2affff072 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:44.239631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d70cf4c-a047-41d3-862e-28a2affff072 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:44.241470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3d70cf4c-a047-41d3-862e-28a2affff072 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:44.305012 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d70cf4c-a047-41d3-862e-28a2affff072 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:44.306601 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d70cf4c-a047-41d3-862e-28a2affff072 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:44.316240 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1242/2499] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:44 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:49.253599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336cf53-853d-4754-a162-19af628a6094 service nova] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:49.254981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-3336cf53-853d-4754-a162-19af628a6094 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:49.310817 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336cf53-853d-4754-a162-19af628a6094 service nova] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:49.314593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e166905-c9d1-4d28-af72-f9da50d243c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:49.315198 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e166905-c9d1-4d28-af72-f9da50d243c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:49.315476 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e166905-c9d1-4d28-af72-f9da50d243c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:49.315748 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e166905-c9d1-4d28-af72-f9da50d243c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:49.317758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9e166905-c9d1-4d28-af72-f9da50d243c3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:49.366173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=100 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:49.376824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e166905-c9d1-4d28-af72-f9da50d243c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:49.378258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e166905-c9d1-4d28-af72-f9da50d243c3 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:49.387220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1258/2500] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:49 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:41:54.311744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=93 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:59.382947 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:41:59.924588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d70cf4c-a047-41d3-862e-28a2affff072 service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:41:59.941651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d70cf4c-a047-41d3-862e-28a2affff072 service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:41:59.947822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a167c2-2257-4b9a-8d47-baaad29142bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:41:59.948470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a167c2-2257-4b9a-8d47-baaad29142bc service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:41:59.948753 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a167c2-2257-4b9a-8d47-baaad29142bc service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:41:59.949033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a167c2-2257-4b9a-8d47-baaad29142bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:41:59.950953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-88a167c2-2257-4b9a-8d47-baaad29142bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:41:59.969689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a167c2-2257-4b9a-8d47-baaad29142bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:41:59.971894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a167c2-2257-4b9a-8d47-baaad29142bc service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:41:59.987918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1243/2501] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:41:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:42:00.016533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a5688-3aad-4a75-a03f-0cc38a955027 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:00.017794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a5688-3aad-4a75-a03f-0cc38a955027 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:00.018489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a5688-3aad-4a75-a03f-0cc38a955027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:00.043669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-803a5688-3aad-4a75-a03f-0cc38a955027 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:42:00.045295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-803a5688-3aad-4a75-a03f-0cc38a955027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:00.117339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1259/2502] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:42:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:42:00.175855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a167c2-2257-4b9a-8d47-baaad29142bc service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:00.177403 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-88a167c2-2257-4b9a-8d47-baaad29142bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:00.248808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a167c2-2257-4b9a-8d47-baaad29142bc service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:00.254480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70ef72-7c40-4996-ae8b-6b29f4b0d2dc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:00.255102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70ef72-7c40-4996-ae8b-6b29f4b0d2dc service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:00.255382 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70ef72-7c40-4996-ae8b-6b29f4b0d2dc service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:00.255655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a70ef72-7c40-4996-ae8b-6b29f4b0d2dc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:00.263626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a70ef72-7c40-4996-ae8b-6b29f4b0d2dc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:00.265254 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a70ef72-7c40-4996-ae8b-6b29f4b0d2dc service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:00.273828 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1244/2503] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:42:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:42:09.980492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803a5688-3aad-4a75-a03f-0cc38a955027 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:09.982771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-803a5688-3aad-4a75-a03f-0cc38a955027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:09.996092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803a5688-3aad-4a75-a03f-0cc38a955027 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:10.000048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af0897c-0cbb-4a3a-bc66-b7487d0e6a89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:10.001441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af0897c-0cbb-4a3a-bc66-b7487d0e6a89 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:10.001940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af0897c-0cbb-4a3a-bc66-b7487d0e6a89 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:10.002313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af0897c-0cbb-4a3a-bc66-b7487d0e6a89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:10.004396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7af0897c-0cbb-4a3a-bc66-b7487d0e6a89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:10.080322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af0897c-0cbb-4a3a-bc66-b7487d0e6a89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:10.081781 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af0897c-0cbb-4a3a-bc66-b7487d0e6a89 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:10.092707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1260/2504] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:42:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:42:10.270640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=38 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:42:13.738037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70ef72-7c40-4996-ae8b-6b29f4b0d2dc service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:13.743637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0a70ef72-7c40-4996-ae8b-6b29f4b0d2dc service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:13.783897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a70ef72-7c40-4996-ae8b-6b29f4b0d2dc service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:13.797195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f65230c-0d2a-4b3b-8cec-44b921a00ffc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:13.798106 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f65230c-0d2a-4b3b-8cec-44b921a00ffc service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:13.798386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f65230c-0d2a-4b3b-8cec-44b921a00ffc service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:13.798662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f65230c-0d2a-4b3b-8cec-44b921a00ffc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:13.800503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4f65230c-0d2a-4b3b-8cec-44b921a00ffc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:13.824756 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f65230c-0d2a-4b3b-8cec-44b921a00ffc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:13.826680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f65230c-0d2a-4b3b-8cec-44b921a00ffc service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:13.845614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1245/2505] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:42:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:42:13.846561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af0897c-0cbb-4a3a-bc66-b7487d0e6a89 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:13.849245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7af0897c-0cbb-4a3a-bc66-b7487d0e6a89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:13.877760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af0897c-0cbb-4a3a-bc66-b7487d0e6a89 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:13.885412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:13.886291 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:13.886612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:13.886974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] PATH_INFO: `/v3/users/fb54ba1355004e14b8dc43cf3c028e67` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:13.888585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb54ba1355004e14b8dc43cf3c028e67)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:13.890907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:13.949475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb54ba1355004e14b8dc43cf3c028e67'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:42:13.958770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb54ba1355004e14b8dc43cf3c028e67'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:42:13.959568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb54ba1355004e14b8dc43cf3c028e67'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:42:13.960247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb54ba1355004e14b8dc43cf3c028e67'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:42:13.974784 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-706992c6-e859-4bb2-a5a2-f081801f114a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb54ba1355004e14b8dc43cf3c028e67'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:42:13.989329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1261/2506] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:42:13 2022] DELETE /identity/v3/users/fb54ba1355004e14b8dc43cf3c028e67 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:42:14.174913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f65230c-0d2a-4b3b-8cec-44b921a00ffc service placement] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:14.198732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f65230c-0d2a-4b3b-8cec-44b921a00ffc service placement] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:14.262874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:14.263605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:14.263985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:14.264261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] PATH_INFO: `/v3/projects/a553921e97994322b71fda7d032ed8bf` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:14.265474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a553921e97994322b71fda7d032ed8bf)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:14.267481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:14.318947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a553921e97994322b71fda7d032ed8bf', 'request_id': 'req-1a598f01-3194-468b-b589-68a18d6e1704'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:42:14.325909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a553921e97994322b71fda7d032ed8bf', 'request_id': 'req-1a598f01-3194-468b-b589-68a18d6e1704'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:42:14.335359 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a553921e97994322b71fda7d032ed8bf', 'request_id': 'req-1a598f01-3194-468b-b589-68a18d6e1704'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:42:14.348329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] The token cache is being invalidate because project a553921e97994322b71fda7d032ed8bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:42:14.348654 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:42:14.349322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1a598f01-3194-468b-b589-68a18d6e1704 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:42:14.351022 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1246/2507] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:42:14 2022] DELETE /identity/v3/projects/a553921e97994322b71fda7d032ed8bf => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:42:14.355779 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee55004-a148-4ec3-9f6e-9074fa96d988 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:14.356165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee55004-a148-4ec3-9f6e-9074fa96d988 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:14.356514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee55004-a148-4ec3-9f6e-9074fa96d988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:14.381432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-eee55004-a148-4ec3-9f6e-9074fa96d988 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:42:14.382956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-eee55004-a148-4ec3-9f6e-9074fa96d988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:14.389113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:42:14.461996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1262/2508] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:42:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:42:14.471723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301f8f67-8af2-4ca2-874a-3c735df94d6b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:14.472011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301f8f67-8af2-4ca2-874a-3c735df94d6b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:14.474116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301f8f67-8af2-4ca2-874a-3c735df94d6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:14.494400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-301f8f67-8af2-4ca2-874a-3c735df94d6b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:42:14.495855 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-301f8f67-8af2-4ca2-874a-3c735df94d6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:14.506680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1247/2509] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:42:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:42:14.510820 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee55004-a148-4ec3-9f6e-9074fa96d988 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:14.520892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee55004-a148-4ec3-9f6e-9074fa96d988 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:14.526659 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bdbec3-0a50-454f-9577-c48628e5c97a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:14.527371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bdbec3-0a50-454f-9577-c48628e5c97a admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:14.527718 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bdbec3-0a50-454f-9577-c48628e5c97a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:14.528061 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bdbec3-0a50-454f-9577-c48628e5c97a admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:14.528835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bdbec3-0a50-454f-9577-c48628e5c97a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:42:14.529210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bdbec3-0a50-454f-9577-c48628e5c97a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:14.530857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bdbec3-0a50-454f-9577-c48628e5c97a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:14.545949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1263/2510] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:42:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:42:14.550599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301f8f67-8af2-4ca2-874a-3c735df94d6b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:14.559822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301f8f67-8af2-4ca2-874a-3c735df94d6b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:14.565102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88c0b5a-d77a-4e9c-ab46-92aeccd4b9fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:14.565709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88c0b5a-d77a-4e9c-ab46-92aeccd4b9fa admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:14.565988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88c0b5a-d77a-4e9c-ab46-92aeccd4b9fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:14.566715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88c0b5a-d77a-4e9c-ab46-92aeccd4b9fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:14.567508 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88c0b5a-d77a-4e9c-ab46-92aeccd4b9fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:14.569316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88c0b5a-d77a-4e9c-ab46-92aeccd4b9fa admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:14.603936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1248/2511] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:42:14 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:42:14.608412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bdbec3-0a50-454f-9577-c48628e5c97a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:14.618520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bdbec3-0a50-454f-9577-c48628e5c97a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:14.680705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aaff155-2543-4cec-8a0a-8205d3b4014b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:14.681328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aaff155-2543-4cec-8a0a-8205d3b4014b admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:14.681586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aaff155-2543-4cec-8a0a-8205d3b4014b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:14.681854 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aaff155-2543-4cec-8a0a-8205d3b4014b admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:14.682484 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aaff155-2543-4cec-8a0a-8205d3b4014b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:14.684397 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aaff155-2543-4cec-8a0a-8205d3b4014b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:14.710912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1264/2512] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:42:14 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:42:14.712203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88c0b5a-d77a-4e9c-ab46-92aeccd4b9fa admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:14.723136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88c0b5a-d77a-4e9c-ab46-92aeccd4b9fa admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:14.729003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b393952d-ba2e-419a-8b2c-575035b8d1a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:14.730390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b393952d-ba2e-419a-8b2c-575035b8d1a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:14.730390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b393952d-ba2e-419a-8b2c-575035b8d1a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:14.730618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b393952d-ba2e-419a-8b2c-575035b8d1a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:14.731398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b393952d-ba2e-419a-8b2c-575035b8d1a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:14.733139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b393952d-ba2e-419a-8b2c-575035b8d1a7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:14.744256 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1249/2513] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:42:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:42:14.748472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aaff155-2543-4cec-8a0a-8205d3b4014b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:14.759431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aaff155-2543-4cec-8a0a-8205d3b4014b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:14.764121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5643427-1cfe-446a-8d1c-1da6d7d89592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:14.764753 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5643427-1cfe-446a-8d1c-1da6d7d89592 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:14.765031 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5643427-1cfe-446a-8d1c-1da6d7d89592 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:14.765301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5643427-1cfe-446a-8d1c-1da6d7d89592 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:14.766231 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5643427-1cfe-446a-8d1c-1da6d7d89592 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:14.767986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5643427-1cfe-446a-8d1c-1da6d7d89592 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:14.779311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1265/2514] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:42:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:42:14.784499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b393952d-ba2e-419a-8b2c-575035b8d1a7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:14.794982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b393952d-ba2e-419a-8b2c-575035b8d1a7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:14.799971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05372db4-b59e-4889-9bd9-d6546f26a415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:14.800649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05372db4-b59e-4889-9bd9-d6546f26a415 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:14.800930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05372db4-b59e-4889-9bd9-d6546f26a415 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:14.801203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05372db4-b59e-4889-9bd9-d6546f26a415 admin admin] PATH_INFO: `/v3/projects/7e247930abd14072b8dfab4043d3e3f5/users/64c98c1da1a847389fa035d86f6326de/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:14.821510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05372db4-b59e-4889-9bd9-d6546f26a415 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e247930abd14072b8dfab4043d3e3f5, user_id=64c98c1da1a847389fa035d86f6326de, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:14.824269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05372db4-b59e-4889-9bd9-d6546f26a415 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:14.837596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1250/2515] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:42:14 2022] PUT /identity/v3/projects/7e247930abd14072b8dfab4043d3e3f5/users/64c98c1da1a847389fa035d86f6326de/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:42:14.842672 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074207d-526f-42a5-ac33-1fd092467adf None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:14.842959 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074207d-526f-42a5-ac33-1fd092467adf None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:14.843228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074207d-526f-42a5-ac33-1fd092467adf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:14.862595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b074207d-526f-42a5-ac33-1fd092467adf None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:42:14.863816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b074207d-526f-42a5-ac33-1fd092467adf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:14.928158 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1266/2516] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:42:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:42:14.958272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05372db4-b59e-4889-9bd9-d6546f26a415 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:14.960400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-05372db4-b59e-4889-9bd9-d6546f26a415 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:14.994621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:42:15.057752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05372db4-b59e-4889-9bd9-d6546f26a415 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:15.063517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052b088c-ef38-425c-bcb4-67d968925f94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:15.065889 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052b088c-ef38-425c-bcb4-67d968925f94 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:15.066430 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052b088c-ef38-425c-bcb4-67d968925f94 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:15.066706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052b088c-ef38-425c-bcb4-67d968925f94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:15.076617 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052b088c-ef38-425c-bcb4-67d968925f94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:15.078070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052b088c-ef38-425c-bcb4-67d968925f94 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:15.088728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1251/2517] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:42:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:42:17.567651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9d8ebd-0953-4d32-834a-f779f0aef5aa None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:17.568152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9d8ebd-0953-4d32-834a-f779f0aef5aa None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:17.568248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9d8ebd-0953-4d32-834a-f779f0aef5aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:17.607756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-6d9d8ebd-0953-4d32-834a-f779f0aef5aa None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:42:17.609563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6d9d8ebd-0953-4d32-834a-f779f0aef5aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:17.635781 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:42:17.703788 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1267/2518] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:42:17 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:42:17.723727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052b088c-ef38-425c-bcb4-67d968925f94 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:17.725263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-052b088c-ef38-425c-bcb4-67d968925f94 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:17.808870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052b088c-ef38-425c-bcb4-67d968925f94 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:17.815602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435b35e-e5c4-4ff0-88ca-7305d8c0b791 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:17.816968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435b35e-e5c4-4ff0-88ca-7305d8c0b791 service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:17.817378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435b35e-e5c4-4ff0-88ca-7305d8c0b791 service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:17.817762 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435b35e-e5c4-4ff0-88ca-7305d8c0b791 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:17.827342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435b35e-e5c4-4ff0-88ca-7305d8c0b791 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:17.830035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435b35e-e5c4-4ff0-88ca-7305d8c0b791 service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:17.847105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1252/2519] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:42:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:42:20.866984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fe25f6-e406-4248-8c0e-f0976fecc2a4 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:20.867588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fe25f6-e406-4248-8c0e-f0976fecc2a4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:20.867588 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fe25f6-e406-4248-8c0e-f0976fecc2a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:20.902996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-41fe25f6-e406-4248-8c0e-f0976fecc2a4 None None] MFA Rules not processed for user `64c98c1da1a847389fa035d86f6326de`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:42:20.911529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-41fe25f6-e406-4248-8c0e-f0976fecc2a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:21.050862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1268/2520] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:42:20 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:42:21.069510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435b35e-e5c4-4ff0-88ca-7305d8c0b791 service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:21.071173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7435b35e-e5c4-4ff0-88ca-7305d8c0b791 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:21.221480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435b35e-e5c4-4ff0-88ca-7305d8c0b791 service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:21.229237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585e2169-76dd-46ad-a1ed-910d528f6ed6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:21.229859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e2169-76dd-46ad-a1ed-910d528f6ed6 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:21.230137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e2169-76dd-46ad-a1ed-910d528f6ed6 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:21.230410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e2169-76dd-46ad-a1ed-910d528f6ed6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:21.241243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585e2169-76dd-46ad-a1ed-910d528f6ed6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:21.243126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585e2169-76dd-46ad-a1ed-910d528f6ed6 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:21.254905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1253/2521] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:42:21 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:42:23.286154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa73f525-c6d7-4ffa-80be-927331670daa None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:23.286691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa73f525-c6d7-4ffa-80be-927331670daa None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:23.286860 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa73f525-c6d7-4ffa-80be-927331670daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:23.309113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-aa73f525-c6d7-4ffa-80be-927331670daa None None] MFA Rules not processed for user `64c98c1da1a847389fa035d86f6326de`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:42:23.310448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-aa73f525-c6d7-4ffa-80be-927331670daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:23.329426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1269/2522] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:42:23 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:42:31.248894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=86 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:42:33.303902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=52 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:42:34.697355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c1fe9-bd4a-4dd1-98e7-e67cefc35b96 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:34.697894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c1fe9-bd4a-4dd1-98e7-e67cefc35b96 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:34.697894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c1fe9-bd4a-4dd1-98e7-e67cefc35b96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:34.717677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-001c1fe9-bd4a-4dd1-98e7-e67cefc35b96 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:42:34.719223 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-001c1fe9-bd4a-4dd1-98e7-e67cefc35b96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:42:34.726255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1254/2523] 10.222.0.25 () {58 vars in 967 bytes} [Mon Sep 26 01:42:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:42:34.738725 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa73f525-c6d7-4ffa-80be-927331670daa None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:42:34.751184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa73f525-c6d7-4ffa-80be-927331670daa None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:42:34.756064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8accbb1-692c-4bb9-9051-c4673765701c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:42:34.756696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8accbb1-692c-4bb9-9051-c4673765701c service swift] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:42:34.756973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8accbb1-692c-4bb9-9051-c4673765701c service swift] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:42:34.757243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8accbb1-692c-4bb9-9051-c4673765701c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:42:34.765830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8accbb1-692c-4bb9-9051-c4673765701c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:42:34.767402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8accbb1-692c-4bb9-9051-c4673765701c service swift] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:42:34.775303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1270/2524] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:42:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:42:44.712246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:42:44.773468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=6 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:01.431707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c1fe9-bd4a-4dd1-98e7-e67cefc35b96 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:01.433139 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-001c1fe9-bd4a-4dd1-98e7-e67cefc35b96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:01.446004 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c1fe9-bd4a-4dd1-98e7-e67cefc35b96 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:01.450755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e0d1f4-6b38-4ae4-8bbb-20f08411ee82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:01.453003 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e0d1f4-6b38-4ae4-8bbb-20f08411ee82 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:01.453307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e0d1f4-6b38-4ae4-8bbb-20f08411ee82 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:01.453628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e0d1f4-6b38-4ae4-8bbb-20f08411ee82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:01.455422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-07e0d1f4-6b38-4ae4-8bbb-20f08411ee82 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:01.467927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e0d1f4-6b38-4ae4-8bbb-20f08411ee82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:01.469392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e0d1f4-6b38-4ae4-8bbb-20f08411ee82 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:01.479090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1255/2525] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:04.523703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8accbb1-692c-4bb9-9051-c4673765701c service swift] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:04.525353 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e8accbb1-692c-4bb9-9051-c4673765701c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:04.588885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e0d1f4-6b38-4ae4-8bbb-20f08411ee82 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:04.590518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-07e0d1f4-6b38-4ae4-8bbb-20f08411ee82 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:04.608143 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e0d1f4-6b38-4ae4-8bbb-20f08411ee82 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:04.612671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:04.613481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:04.614006 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:04.614473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] PATH_INFO: `/v3/users/06043e862a394764a58003f205cbb484` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:04.615784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] RBAC: Authorizing `identity:delete_user(user_id=06043e862a394764a58003f205cbb484)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:04.617829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:04.626060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8accbb1-692c-4bb9-9051-c4673765701c service swift] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:04.631592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d32bfb-95cd-4c25-928c-f4df42743aa2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:04.632201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d32bfb-95cd-4c25-928c-f4df42743aa2 service placement] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:04.632515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d32bfb-95cd-4c25-928c-f4df42743aa2 service placement] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:04.632790 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d32bfb-95cd-4c25-928c-f4df42743aa2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:04.634577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-14d32bfb-95cd-4c25-928c-f4df42743aa2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:04.646578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d32bfb-95cd-4c25-928c-f4df42743aa2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:04.648163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d32bfb-95cd-4c25-928c-f4df42743aa2 service placement] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:04.659183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1271/2526] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:04.677896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '06043e862a394764a58003f205cbb484'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:04.685783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '06043e862a394764a58003f205cbb484'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:04.686420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '06043e862a394764a58003f205cbb484'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:04.687049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '06043e862a394764a58003f205cbb484'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:04.695494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a914acb0-683c-460a-91dd-3c78b7f45781 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '06043e862a394764a58003f205cbb484'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:04.714176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1256/2527] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:04 2022] DELETE /identity/v3/users/06043e862a394764a58003f205cbb484 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:04.887358 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d32bfb-95cd-4c25-928c-f4df42743aa2 service placement] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:04.899119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d32bfb-95cd-4c25-928c-f4df42743aa2 service placement] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:04.958417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:04.959045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:04.959327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:04.959670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] PATH_INFO: `/v3/projects/3677555d86804f03ae243765cfd3fd24` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:04.960829 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3677555d86804f03ae243765cfd3fd24)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:04.962791 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:05.010497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3677555d86804f03ae243765cfd3fd24', 'request_id': 'req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:05.017041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3677555d86804f03ae243765cfd3fd24', 'request_id': 'req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:05.022487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3677555d86804f03ae243765cfd3fd24', 'request_id': 'req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:05.035427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] The token cache is being invalidate because project 3677555d86804f03ae243765cfd3fd24 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:05.035721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:05.036387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e8600372-f565-41e0-bbb5-c8ed4cd7ba01 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:05.037916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1272/2528] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:04 2022] DELETE /identity/v3/projects/3677555d86804f03ae243765cfd3fd24 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:05.042947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d00ebc9-f738-48c3-84e5-ce19181fb4d1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:05.043237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d00ebc9-f738-48c3-84e5-ce19181fb4d1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:05.043514 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d00ebc9-f738-48c3-84e5-ce19181fb4d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:05.063660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7d00ebc9-f738-48c3-84e5-ce19181fb4d1 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:05.066020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7d00ebc9-f738-48c3-84e5-ce19181fb4d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:05.124705 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1257/2529] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:05.133476 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b61daf0-6532-4ba9-8854-64dd76c12b80 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:05.133757 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b61daf0-6532-4ba9-8854-64dd76c12b80 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:05.134028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b61daf0-6532-4ba9-8854-64dd76c12b80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:05.153512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0b61daf0-6532-4ba9-8854-64dd76c12b80 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:05.154710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0b61daf0-6532-4ba9-8854-64dd76c12b80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:05.162649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1273/2530] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:05.166992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d00ebc9-f738-48c3-84e5-ce19181fb4d1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:05.177791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d00ebc9-f738-48c3-84e5-ce19181fb4d1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:05.182320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9a9c83-59ab-42f1-9392-66fe242b3266 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:05.182929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9a9c83-59ab-42f1-9392-66fe242b3266 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:05.183208 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9a9c83-59ab-42f1-9392-66fe242b3266 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:05.183476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9a9c83-59ab-42f1-9392-66fe242b3266 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:05.184137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9a9c83-59ab-42f1-9392-66fe242b3266 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:05.184480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9a9c83-59ab-42f1-9392-66fe242b3266 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:05.186068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9a9c83-59ab-42f1-9392-66fe242b3266 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:05.198130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1258/2531] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:43:05.202370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b61daf0-6532-4ba9-8854-64dd76c12b80 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:05.212025 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b61daf0-6532-4ba9-8854-64dd76c12b80 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:05.216698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cbe780-6251-4d34-9b71-63ada62ec604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:05.217285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cbe780-6251-4d34-9b71-63ada62ec604 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:05.217564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cbe780-6251-4d34-9b71-63ada62ec604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:05.217840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cbe780-6251-4d34-9b71-63ada62ec604 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:05.218466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cbe780-6251-4d34-9b71-63ada62ec604 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:05.220361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cbe780-6251-4d34-9b71-63ada62ec604 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:05.231671 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:05.248807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1274/2532] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:05 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:05.253137 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9a9c83-59ab-42f1-9392-66fe242b3266 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:05.262953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9a9c83-59ab-42f1-9392-66fe242b3266 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:05.307371 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:05.322901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0079fd21-7fb1-45c5-bb58-546e6e8ee4fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:05.323502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0079fd21-7fb1-45c5-bb58-546e6e8ee4fa admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:05.323784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0079fd21-7fb1-45c5-bb58-546e6e8ee4fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:05.324058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0079fd21-7fb1-45c5-bb58-546e6e8ee4fa admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:05.324728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0079fd21-7fb1-45c5-bb58-546e6e8ee4fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:05.326683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0079fd21-7fb1-45c5-bb58-546e6e8ee4fa admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:05.347971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1259/2533] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:05 2022] POST /identity/v3/users => generated 287 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:05.352230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cbe780-6251-4d34-9b71-63ada62ec604 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:05.362577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cbe780-6251-4d34-9b71-63ada62ec604 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:05.367158 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661b2e66-3895-44ac-b395-0f5ce5e111b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:05.367746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661b2e66-3895-44ac-b395-0f5ce5e111b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:05.368067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661b2e66-3895-44ac-b395-0f5ce5e111b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:05.368300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661b2e66-3895-44ac-b395-0f5ce5e111b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:05.368985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661b2e66-3895-44ac-b395-0f5ce5e111b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:05.370732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661b2e66-3895-44ac-b395-0f5ce5e111b7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:05.381040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1275/2534] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:05 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:05.385078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0079fd21-7fb1-45c5-bb58-546e6e8ee4fa admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:05.395859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0079fd21-7fb1-45c5-bb58-546e6e8ee4fa admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:05.400564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb31f18-6bb1-4e15-9c61-d372bedfae57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:05.401157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb31f18-6bb1-4e15-9c61-d372bedfae57 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:05.401440 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb31f18-6bb1-4e15-9c61-d372bedfae57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:05.401720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb31f18-6bb1-4e15-9c61-d372bedfae57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:05.402361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb31f18-6bb1-4e15-9c61-d372bedfae57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:05.403965 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb31f18-6bb1-4e15-9c61-d372bedfae57 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:05.415067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1260/2535] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:05.419274 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661b2e66-3895-44ac-b395-0f5ce5e111b7 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:05.429366 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661b2e66-3895-44ac-b395-0f5ce5e111b7 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:05.434087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc89da1-11cc-466c-9049-229c7dbd2202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:05.434727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc89da1-11cc-466c-9049-229c7dbd2202 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:05.435007 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc89da1-11cc-466c-9049-229c7dbd2202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:05.435284 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc89da1-11cc-466c-9049-229c7dbd2202 admin admin] PATH_INFO: `/v3/projects/341bfb664e114126b855c74373c93c0f/users/af6fc1d42016450ca9372e4f2ed6ce98/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:05.450378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc89da1-11cc-466c-9049-229c7dbd2202 admin admin] RBAC: Authorizing `identity:create_grant(project_id=341bfb664e114126b855c74373c93c0f, user_id=af6fc1d42016450ca9372e4f2ed6ce98, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:05.453078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc89da1-11cc-466c-9049-229c7dbd2202 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:05.467452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1276/2536] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:05 2022] PUT /identity/v3/projects/341bfb664e114126b855c74373c93c0f/users/af6fc1d42016450ca9372e4f2ed6ce98/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:05.478301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad52cc-9bb2-4cf0-82c3-5044b7fa3cb0 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:05.478614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad52cc-9bb2-4cf0-82c3-5044b7fa3cb0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:05.478896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad52cc-9bb2-4cf0-82c3-5044b7fa3cb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:05.507094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-06ad52cc-9bb2-4cf0-82c3-5044b7fa3cb0 None None] MFA Rules not processed for user `af6fc1d42016450ca9372e4f2ed6ce98`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:05.508312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-06ad52cc-9bb2-4cf0-82c3-5044b7fa3cb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:05.572817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1261/2537] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:05 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:05.585711 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc89da1-11cc-466c-9049-229c7dbd2202 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:05.587479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5cc89da1-11cc-466c-9049-229c7dbd2202 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:05.662603 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc89da1-11cc-466c-9049-229c7dbd2202 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:05.668169 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b3b610-5a63-4a9f-ae7d-7c70e9037b40 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:05.668799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b3b610-5a63-4a9f-ae7d-7c70e9037b40 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:05.669078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b3b610-5a63-4a9f-ae7d-7c70e9037b40 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:05.669347 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b3b610-5a63-4a9f-ae7d-7c70e9037b40 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:05.677036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b3b610-5a63-4a9f-ae7d-7c70e9037b40 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:05.678615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b3b610-5a63-4a9f-ae7d-7c70e9037b40 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:05.687240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1277/2538] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:05 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:08.360551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ad52cc-9bb2-4cf0-82c3-5044b7fa3cb0 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:08.421592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ad52cc-9bb2-4cf0-82c3-5044b7fa3cb0 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:08.426864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:08.427485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:08.427765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:08.428037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] PATH_INFO: `/v3/users/af6fc1d42016450ca9372e4f2ed6ce98` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:08.429204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=af6fc1d42016450ca9372e4f2ed6ce98)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:08.431124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:08.475705 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af6fc1d42016450ca9372e4f2ed6ce98'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:08.483051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af6fc1d42016450ca9372e4f2ed6ce98'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:08.483677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af6fc1d42016450ca9372e4f2ed6ce98'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:08.484306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af6fc1d42016450ca9372e4f2ed6ce98'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:08.486915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:08.493647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af6fc1d42016450ca9372e4f2ed6ce98'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:08.511030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1262/2539] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:08 2022] DELETE /identity/v3/users/af6fc1d42016450ca9372e4f2ed6ce98 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:08.515441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ca681-8028-427b-8e5f-cfbc4de1fc49 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:08.515900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ca681-8028-427b-8e5f-cfbc4de1fc49 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:08.516002 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ca681-8028-427b-8e5f-cfbc4de1fc49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:08.536618 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b89ca681-8028-427b-8e5f-cfbc4de1fc49 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:08.537831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b89ca681-8028-427b-8e5f-cfbc4de1fc49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:08.599387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1278/2540] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:08.609370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:08.610873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:08.668446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1746983-e6a7-4d7b-96d5-23718a195c1a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:08.672020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8964b7fd-5c09-4c67-80b3-e171761fb097 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:08.672637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8964b7fd-5c09-4c67-80b3-e171761fb097 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:08.672918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8964b7fd-5c09-4c67-80b3-e171761fb097 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:08.673187 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8964b7fd-5c09-4c67-80b3-e171761fb097 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:08.683008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8964b7fd-5c09-4c67-80b3-e171761fb097 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:08.684505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8964b7fd-5c09-4c67-80b3-e171761fb097 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:08.695022 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1263/2541] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:08.966341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89ca681-8028-427b-8e5f-cfbc4de1fc49 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:08.967800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b89ca681-8028-427b-8e5f-cfbc4de1fc49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:08.983401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89ca681-8028-427b-8e5f-cfbc4de1fc49 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:08.988615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:08.989481 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:08.989991 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:08.990273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] PATH_INFO: `/v3/projects/341bfb664e114126b855c74373c93c0f` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:08.991421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=341bfb664e114126b855c74373c93c0f)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:08.993478 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.043042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:09.051387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '341bfb664e114126b855c74373c93c0f', 'request_id': 'req-4b74d1fe-896c-4597-9c1a-18b65b7384d0'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:09.058950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '341bfb664e114126b855c74373c93c0f', 'request_id': 'req-4b74d1fe-896c-4597-9c1a-18b65b7384d0'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:09.066539 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '341bfb664e114126b855c74373c93c0f', 'request_id': 'req-4b74d1fe-896c-4597-9c1a-18b65b7384d0'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:09.083349 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] The token cache is being invalidate because project 341bfb664e114126b855c74373c93c0f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:09.083685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:09.084498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4b74d1fe-896c-4597-9c1a-18b65b7384d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:09.086129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1279/2542] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:08 2022] DELETE /identity/v3/projects/341bfb664e114126b855c74373c93c0f => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:09.090909 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736a403e-d8ed-4f84-a303-1acee6bdb140 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.091200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736a403e-d8ed-4f84-a303-1acee6bdb140 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.091476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736a403e-d8ed-4f84-a303-1acee6bdb140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.113223 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-736a403e-d8ed-4f84-a303-1acee6bdb140 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:09.114362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-736a403e-d8ed-4f84-a303-1acee6bdb140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:09.174897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1264/2543] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:09.183555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc839bfa-409a-492a-9200-24160409289b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.183840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc839bfa-409a-492a-9200-24160409289b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.184120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc839bfa-409a-492a-9200-24160409289b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.204230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-fc839bfa-409a-492a-9200-24160409289b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:09.205533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fc839bfa-409a-492a-9200-24160409289b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:09.213640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1280/2544] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:09.217908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736a403e-d8ed-4f84-a303-1acee6bdb140 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.228579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736a403e-d8ed-4f84-a303-1acee6bdb140 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.233163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b785aea-3143-4693-b62a-fef1a681aad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:09.233757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b785aea-3143-4693-b62a-fef1a681aad8 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.234036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b785aea-3143-4693-b62a-fef1a681aad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.234311 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b785aea-3143-4693-b62a-fef1a681aad8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.234954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b785aea-3143-4693-b62a-fef1a681aad8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:09.235386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b785aea-3143-4693-b62a-fef1a681aad8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:09.237129 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b785aea-3143-4693-b62a-fef1a681aad8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.248625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1265/2545] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:43:09.253750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc839bfa-409a-492a-9200-24160409289b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.263236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc839bfa-409a-492a-9200-24160409289b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.268472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60de8ff-b870-429f-a3cf-f7f3cd19a1ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:09.269056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60de8ff-b870-429f-a3cf-f7f3cd19a1ff admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.269331 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60de8ff-b870-429f-a3cf-f7f3cd19a1ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.269598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60de8ff-b870-429f-a3cf-f7f3cd19a1ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.270216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60de8ff-b870-429f-a3cf-f7f3cd19a1ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:09.272104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60de8ff-b870-429f-a3cf-f7f3cd19a1ff admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.299651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1281/2546] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:09 2022] POST /identity/v3/projects => generated 365 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:09.303830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b785aea-3143-4693-b62a-fef1a681aad8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.313693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b785aea-3143-4693-b62a-fef1a681aad8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.369746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a763436-9ad0-4763-bd7c-4d7e11d2f356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:09.370390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a763436-9ad0-4763-bd7c-4d7e11d2f356 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.370537 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a763436-9ad0-4763-bd7c-4d7e11d2f356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.370810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a763436-9ad0-4763-bd7c-4d7e11d2f356 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.371444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a763436-9ad0-4763-bd7c-4d7e11d2f356 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:09.373392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a763436-9ad0-4763-bd7c-4d7e11d2f356 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.383087 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:09.395905 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1266/2547] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:09 2022] POST /identity/v3/users => generated 283 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:09.400050 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60de8ff-b870-429f-a3cf-f7f3cd19a1ff admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.411393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60de8ff-b870-429f-a3cf-f7f3cd19a1ff admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.416251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e11a3a0-c3c8-499d-860b-ad038bbd8402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:09.416942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e11a3a0-c3c8-499d-860b-ad038bbd8402 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.417220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e11a3a0-c3c8-499d-860b-ad038bbd8402 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.417504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e11a3a0-c3c8-499d-860b-ad038bbd8402 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.418138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e11a3a0-c3c8-499d-860b-ad038bbd8402 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:09.419737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e11a3a0-c3c8-499d-860b-ad038bbd8402 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.430300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1282/2548] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:09.434216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a763436-9ad0-4763-bd7c-4d7e11d2f356 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.444461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a763436-9ad0-4763-bd7c-4d7e11d2f356 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.449041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7476111b-0bd2-459a-92f1-d69d2d82b7bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:09.449626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7476111b-0bd2-459a-92f1-d69d2d82b7bb admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.449907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7476111b-0bd2-459a-92f1-d69d2d82b7bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.450179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7476111b-0bd2-459a-92f1-d69d2d82b7bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.450816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7476111b-0bd2-459a-92f1-d69d2d82b7bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:09.452461 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7476111b-0bd2-459a-92f1-d69d2d82b7bb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.463606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1267/2549] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:09.467551 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e11a3a0-c3c8-499d-860b-ad038bbd8402 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.478084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e11a3a0-c3c8-499d-860b-ad038bbd8402 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.482643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff98a22-af88-459f-9cad-6f5dd08c2dce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:09.483323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff98a22-af88-459f-9cad-6f5dd08c2dce admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.483598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff98a22-af88-459f-9cad-6f5dd08c2dce admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.483870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff98a22-af88-459f-9cad-6f5dd08c2dce admin admin] PATH_INFO: `/v3/projects/fe3bb8316d054416a58f3d082ffeb3d1/users/e04fc28dee25421f8087b7c8b5a675ea/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.499249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff98a22-af88-459f-9cad-6f5dd08c2dce admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe3bb8316d054416a58f3d082ffeb3d1, user_id=e04fc28dee25421f8087b7c8b5a675ea, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:09.502310 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff98a22-af88-459f-9cad-6f5dd08c2dce admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.515408 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1283/2550] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:09 2022] PUT /identity/v3/projects/fe3bb8316d054416a58f3d082ffeb3d1/users/e04fc28dee25421f8087b7c8b5a675ea/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:09.527764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2a5e7a-9212-4bf9-986a-37c290dddfe7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.528056 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2a5e7a-9212-4bf9-986a-37c290dddfe7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.528333 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2a5e7a-9212-4bf9-986a-37c290dddfe7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.558233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fa2a5e7a-9212-4bf9-986a-37c290dddfe7 None None] MFA Rules not processed for user `e04fc28dee25421f8087b7c8b5a675ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:09.559445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fa2a5e7a-9212-4bf9-986a-37c290dddfe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:09.624091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1268/2551] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:09 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:09.628774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff98a22-af88-459f-9cad-6f5dd08c2dce admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.694775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff98a22-af88-459f-9cad-6f5dd08c2dce admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.699608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed4486e-4df6-4e7e-93ec-971d78e08662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:09.700239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed4486e-4df6-4e7e-93ec-971d78e08662 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.700561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed4486e-4df6-4e7e-93ec-971d78e08662 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.700834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed4486e-4df6-4e7e-93ec-971d78e08662 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.701513 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed4486e-4df6-4e7e-93ec-971d78e08662 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:09.703487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed4486e-4df6-4e7e-93ec-971d78e08662 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.738051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1284/2552] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:09 2022] POST /identity/v3/projects => generated 363 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:09.739607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2a5e7a-9212-4bf9-986a-37c290dddfe7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.751146 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2a5e7a-9212-4bf9-986a-37c290dddfe7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.814506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f609d219-ecfd-4ba9-b322-5f60abee1fcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:09.815376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f609d219-ecfd-4ba9-b322-5f60abee1fcf admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.815656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f609d219-ecfd-4ba9-b322-5f60abee1fcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.815923 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f609d219-ecfd-4ba9-b322-5f60abee1fcf admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.816614 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f609d219-ecfd-4ba9-b322-5f60abee1fcf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:09.818570 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f609d219-ecfd-4ba9-b322-5f60abee1fcf admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.841218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1269/2553] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:09 2022] POST /identity/v3/users => generated 288 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:09.845266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed4486e-4df6-4e7e-93ec-971d78e08662 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.856015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed4486e-4df6-4e7e-93ec-971d78e08662 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.860679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e7d742-28ff-4b78-8d8c-1d4e5c00f8af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:09.861529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e7d742-28ff-4b78-8d8c-1d4e5c00f8af admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.861808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e7d742-28ff-4b78-8d8c-1d4e5c00f8af admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.862107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e7d742-28ff-4b78-8d8c-1d4e5c00f8af admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.862752 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e7d742-28ff-4b78-8d8c-1d4e5c00f8af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:09.864467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e7d742-28ff-4b78-8d8c-1d4e5c00f8af admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.875268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1285/2554] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:09.879958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f609d219-ecfd-4ba9-b322-5f60abee1fcf admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.890906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f609d219-ecfd-4ba9-b322-5f60abee1fcf admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.902154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f12fde9-c00f-4f23-84ed-8721b4d341c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:09.904117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f12fde9-c00f-4f23-84ed-8721b4d341c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:09.904435 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f12fde9-c00f-4f23-84ed-8721b4d341c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:09.904712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f12fde9-c00f-4f23-84ed-8721b4d341c0 admin admin] PATH_INFO: `/v3/projects/f329308292794339ac539363375ccfa9/users/0fe10f3475e8460f9fefbb75fc2f4712/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:09.921532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f12fde9-c00f-4f23-84ed-8721b4d341c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f329308292794339ac539363375ccfa9, user_id=0fe10f3475e8460f9fefbb75fc2f4712, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:09.924370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f12fde9-c00f-4f23-84ed-8721b4d341c0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:09.945752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1270/2555] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:09 2022] PUT /identity/v3/projects/f329308292794339ac539363375ccfa9/users/0fe10f3475e8460f9fefbb75fc2f4712/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:09.949881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e7d742-28ff-4b78-8d8c-1d4e5c00f8af admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:09.960600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e7d742-28ff-4b78-8d8c-1d4e5c00f8af admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:09.992083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:10.022282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08d8c3c-b2cf-478d-8e9a-5f129fedaf4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:10.023212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d8c3c-b2cf-478d-8e9a-5f129fedaf4d admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:10.023212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d8c3c-b2cf-478d-8e9a-5f129fedaf4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:10.023486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d8c3c-b2cf-478d-8e9a-5f129fedaf4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:10.024280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08d8c3c-b2cf-478d-8e9a-5f129fedaf4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:10.025965 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08d8c3c-b2cf-478d-8e9a-5f129fedaf4d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:10.037567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1286/2556] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:09 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:10.041725 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f12fde9-c00f-4f23-84ed-8721b4d341c0 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:10.052605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f12fde9-c00f-4f23-84ed-8721b4d341c0 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:10.057831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353c62c6-df9a-467d-9712-c82c69bf3ea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:10.058653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353c62c6-df9a-467d-9712-c82c69bf3ea7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:10.058969 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353c62c6-df9a-467d-9712-c82c69bf3ea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:10.059420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353c62c6-df9a-467d-9712-c82c69bf3ea7 admin admin] PATH_INFO: `/v3/projects/f329308292794339ac539363375ccfa9/users/0fe10f3475e8460f9fefbb75fc2f4712/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:10.062784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353c62c6-df9a-467d-9712-c82c69bf3ea7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f329308292794339ac539363375ccfa9, user_id=0fe10f3475e8460f9fefbb75fc2f4712, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:10.066682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353c62c6-df9a-467d-9712-c82c69bf3ea7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:10.078799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1271/2557] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:10 2022] PUT /identity/v3/projects/f329308292794339ac539363375ccfa9/users/0fe10f3475e8460f9fefbb75fc2f4712/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:10.089817 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4283f2-8be8-4d4c-8f88-69b1a897f398 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:10.090104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4283f2-8be8-4d4c-8f88-69b1a897f398 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:10.090378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4283f2-8be8-4d4c-8f88-69b1a897f398 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:10.120431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0e4283f2-8be8-4d4c-8f88-69b1a897f398 None None] MFA Rules not processed for user `0fe10f3475e8460f9fefbb75fc2f4712`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:10.121651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0e4283f2-8be8-4d4c-8f88-69b1a897f398 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:10.192469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1287/2558] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:10 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:10.203721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353c62c6-df9a-467d-9712-c82c69bf3ea7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:10.206295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-353c62c6-df9a-467d-9712-c82c69bf3ea7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:10.234513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:10.267302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353c62c6-df9a-467d-9712-c82c69bf3ea7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:10.270301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7c563-ef68-4fe5-aac5-24321a0a75b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:10.270911 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7c563-ef68-4fe5-aac5-24321a0a75b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:10.271192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7c563-ef68-4fe5-aac5-24321a0a75b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:10.271467 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7c563-ef68-4fe5-aac5-24321a0a75b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:10.280685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea7c563-ef68-4fe5-aac5-24321a0a75b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:10.282122 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea7c563-ef68-4fe5-aac5-24321a0a75b0 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:10.291895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1272/2559] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:10 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:10.368293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4283f2-8be8-4d4c-8f88-69b1a897f398 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:10.378689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4283f2-8be8-4d4c-8f88-69b1a897f398 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:10.382296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655a966f-9833-4211-948c-540c3eff5a9a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:10.382886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655a966f-9833-4211-948c-540c3eff5a9a service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:10.383164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655a966f-9833-4211-948c-540c3eff5a9a service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:10.383435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655a966f-9833-4211-948c-540c3eff5a9a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:10.440435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655a966f-9833-4211-948c-540c3eff5a9a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:10.441798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655a966f-9833-4211-948c-540c3eff5a9a service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:10.450762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1288/2560] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:10 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:18.778569 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7c563-ef68-4fe5-aac5-24321a0a75b0 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:18.779996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-aea7c563-ef68-4fe5-aac5-24321a0a75b0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:18.839484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7c563-ef68-4fe5-aac5-24321a0a75b0 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:18.843214 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff80368a-0c0c-4194-886d-469c56cacbce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:18.843814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80368a-0c0c-4194-886d-469c56cacbce service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:18.844088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80368a-0c0c-4194-886d-469c56cacbce service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:18.844370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80368a-0c0c-4194-886d-469c56cacbce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:18.846968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ff80368a-0c0c-4194-886d-469c56cacbce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:18.913054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff80368a-0c0c-4194-886d-469c56cacbce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:18.914437 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff80368a-0c0c-4194-886d-469c56cacbce service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:18.925048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1273/2561] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:20.446521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=75 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:20.943712 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655a966f-9833-4211-948c-540c3eff5a9a service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:20.955146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655a966f-9833-4211-948c-540c3eff5a9a service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:20.959938 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:20.960710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:20.961162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:20.961496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] PATH_INFO: `/v3/users/e04fc28dee25421f8087b7c8b5a675ea` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:20.962692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e04fc28dee25421f8087b7c8b5a675ea)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:20.964742 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:21.012841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e04fc28dee25421f8087b7c8b5a675ea'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.021246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e04fc28dee25421f8087b7c8b5a675ea'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.021885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e04fc28dee25421f8087b7c8b5a675ea'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.022504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e04fc28dee25421f8087b7c8b5a675ea'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.041810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e04fc28dee25421f8087b7c8b5a675ea'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.054402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1289/2562] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:20 2022] DELETE /identity/v3/users/e04fc28dee25421f8087b7c8b5a675ea => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:21.058961 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743a373c-4155-4e05-8159-ccfca91b89bc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:21.059400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743a373c-4155-4e05-8159-ccfca91b89bc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:21.059567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743a373c-4155-4e05-8159-ccfca91b89bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:21.086750 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-743a373c-4155-4e05-8159-ccfca91b89bc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:21.087954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-743a373c-4155-4e05-8159-ccfca91b89bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:21.149785 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1274/2563] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:21.159210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:21.160686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:21.221360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16c7769-4512-4cf1-b1b2-84077f1a6165 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:21.225555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ff637c-f288-4eb6-9ae0-bed4f8bcbd77 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:21.226634 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ff637c-f288-4eb6-9ae0-bed4f8bcbd77 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:21.226786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ff637c-f288-4eb6-9ae0-bed4f8bcbd77 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:21.226934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ff637c-f288-4eb6-9ae0-bed4f8bcbd77 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:21.237205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ff637c-f288-4eb6-9ae0-bed4f8bcbd77 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:21.238668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ff637c-f288-4eb6-9ae0-bed4f8bcbd77 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:21.249480 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1290/2564] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:21.583293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743a373c-4155-4e05-8159-ccfca91b89bc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:21.585069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-743a373c-4155-4e05-8159-ccfca91b89bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:21.604736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743a373c-4155-4e05-8159-ccfca91b89bc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:21.609496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:21.610265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:21.610605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:21.611511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] PATH_INFO: `/v3/projects/fe3bb8316d054416a58f3d082ffeb3d1` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:21.612870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe3bb8316d054416a58f3d082ffeb3d1)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:21.614814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:21.627741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:21.693394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe3bb8316d054416a58f3d082ffeb3d1', 'request_id': 'req-60e06978-074d-4acb-a2c9-53b36b295b1e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.701972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe3bb8316d054416a58f3d082ffeb3d1', 'request_id': 'req-60e06978-074d-4acb-a2c9-53b36b295b1e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.712760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe3bb8316d054416a58f3d082ffeb3d1', 'request_id': 'req-60e06978-074d-4acb-a2c9-53b36b295b1e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.724730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] The token cache is being invalidate because project fe3bb8316d054416a58f3d082ffeb3d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:21.725111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.725922 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.727846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1275/2565] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:21 2022] DELETE /identity/v3/projects/fe3bb8316d054416a58f3d082ffeb3d1 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:21.731778 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ff637c-f288-4eb6-9ae0-bed4f8bcbd77 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:21.745624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ff637c-f288-4eb6-9ae0-bed4f8bcbd77 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:21.830450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:21.831175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:21.831461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:21.831733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] PATH_INFO: `/v3/users/0fe10f3475e8460f9fefbb75fc2f4712` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:21.833022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0fe10f3475e8460f9fefbb75fc2f4712)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:21.835286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:21.855338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:21.904704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0fe10f3475e8460f9fefbb75fc2f4712'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.915065 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0fe10f3475e8460f9fefbb75fc2f4712'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.915793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0fe10f3475e8460f9fefbb75fc2f4712'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.916602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0fe10f3475e8460f9fefbb75fc2f4712'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.942887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0fe10f3475e8460f9fefbb75fc2f4712'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:21.960495 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1291/2566] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:21 2022] DELETE /identity/v3/users/0fe10f3475e8460f9fefbb75fc2f4712 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:22.194929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.196387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:22.209926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e06978-074d-4acb-a2c9-53b36b295b1e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.269832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.270668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.271045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.271327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] PATH_INFO: `/v3/users/bc62747b8aab40d9bff9d853b9b73199` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.272588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc62747b8aab40d9bff9d853b9b73199)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.274703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.293941 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.295535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:22.311136 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aadfa9-179d-4265-846a-c7d4978e1a81 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.318566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.319578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.319926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.320287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] PATH_INFO: `/v3/projects/f329308292794339ac539363375ccfa9` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.324272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f329308292794339ac539363375ccfa9)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.326944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.337657 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc62747b8aab40d9bff9d853b9b73199'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.346226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc62747b8aab40d9bff9d853b9b73199'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.346850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc62747b8aab40d9bff9d853b9b73199'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.347484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc62747b8aab40d9bff9d853b9b73199'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.355507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-90436f1d-a648-4dd6-beae-06bd255e0a15 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc62747b8aab40d9bff9d853b9b73199'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.374182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1276/2567] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:22 2022] DELETE /identity/v3/users/bc62747b8aab40d9bff9d853b9b73199 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:22.379195 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f329308292794339ac539363375ccfa9', 'request_id': 'req-60167b78-9a1d-4441-8bba-d76b9a85d472'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.385833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f329308292794339ac539363375ccfa9', 'request_id': 'req-60167b78-9a1d-4441-8bba-d76b9a85d472'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.394468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f329308292794339ac539363375ccfa9', 'request_id': 'req-60167b78-9a1d-4441-8bba-d76b9a85d472'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.407914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] The token cache is being invalidate because project f329308292794339ac539363375ccfa9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:22.408427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.409220 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-60167b78-9a1d-4441-8bba-d76b9a85d472 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.414349 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1292/2568] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:22 2022] DELETE /identity/v3/projects/f329308292794339ac539363375ccfa9 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:22.415946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cedd87-4b1a-4cfa-a3ca-3be61a253bca None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.416235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cedd87-4b1a-4cfa-a3ca-3be61a253bca None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.416540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cedd87-4b1a-4cfa-a3ca-3be61a253bca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.437732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b6cedd87-4b1a-4cfa-a3ca-3be61a253bca None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:22.439079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b6cedd87-4b1a-4cfa-a3ca-3be61a253bca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:22.452500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:22.503689 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1277/2569] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:22.511359 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaeee90-54a4-4a30-ab5b-996100ba9842 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.511649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaeee90-54a4-4a30-ab5b-996100ba9842 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.511960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaeee90-54a4-4a30-ab5b-996100ba9842 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.537509 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-cdaeee90-54a4-4a30-ab5b-996100ba9842 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:22.538738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-cdaeee90-54a4-4a30-ab5b-996100ba9842 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:22.548106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1293/2570] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:22.552462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cedd87-4b1a-4cfa-a3ca-3be61a253bca None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.564212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cedd87-4b1a-4cfa-a3ca-3be61a253bca None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.569745 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ba2cc4-88ba-4109-a205-6ef414eb923a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.570348 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ba2cc4-88ba-4109-a205-6ef414eb923a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.570624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ba2cc4-88ba-4109-a205-6ef414eb923a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.570984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ba2cc4-88ba-4109-a205-6ef414eb923a admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.571650 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ba2cc4-88ba-4109-a205-6ef414eb923a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:22.571952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ba2cc4-88ba-4109-a205-6ef414eb923a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.573592 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ba2cc4-88ba-4109-a205-6ef414eb923a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.574491 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaeee90-54a4-4a30-ab5b-996100ba9842 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.577847 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-cdaeee90-54a4-4a30-ab5b-996100ba9842 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:22.590650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaeee90-54a4-4a30-ab5b-996100ba9842 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.596261 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.596961 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.597243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.597545 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] PATH_INFO: `/v3/projects/2f78ca6e22f04c4ab911b2e308338517` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.598688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2f78ca6e22f04c4ab911b2e308338517)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.600646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.607046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1278/2571] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:43:22.612176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ba2cc4-88ba-4109-a205-6ef414eb923a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.633512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ba2cc4-88ba-4109-a205-6ef414eb923a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.638605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53934f6-b789-4ea2-8100-ec69e635c8ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.639684 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53934f6-b789-4ea2-8100-ec69e635c8ed admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.640048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53934f6-b789-4ea2-8100-ec69e635c8ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.640559 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53934f6-b789-4ea2-8100-ec69e635c8ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.641339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53934f6-b789-4ea2-8100-ec69e635c8ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.643906 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53934f6-b789-4ea2-8100-ec69e635c8ed admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.655058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2f78ca6e22f04c4ab911b2e308338517', 'request_id': 'req-21fca6c2-66db-4412-beda-4e47065d0e11'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.662376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2f78ca6e22f04c4ab911b2e308338517', 'request_id': 'req-21fca6c2-66db-4412-beda-4e47065d0e11'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.669747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2f78ca6e22f04c4ab911b2e308338517', 'request_id': 'req-21fca6c2-66db-4412-beda-4e47065d0e11'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.673730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:DELETE=22,keystone:SELECT=75,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:22.678180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1279/2572] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:22 2022] POST /identity/v3/projects => generated 365 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:43:22.682952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53934f6-b789-4ea2-8100-ec69e635c8ed admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.687003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] The token cache is being invalidate because project 2f78ca6e22f04c4ab911b2e308338517 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:22.687483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.688170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21fca6c2-66db-4412-beda-4e47065d0e11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:22.693578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1294/2573] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:22 2022] DELETE /identity/v3/projects/2f78ca6e22f04c4ab911b2e308338517 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:22.694949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372c36ac-d58a-4582-a5bf-3f7d37353254 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.695344 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372c36ac-d58a-4582-a5bf-3f7d37353254 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.695623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372c36ac-d58a-4582-a5bf-3f7d37353254 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.716615 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-372c36ac-d58a-4582-a5bf-3f7d37353254 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:22.717893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-372c36ac-d58a-4582-a5bf-3f7d37353254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:22.749483 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53934f6-b789-4ea2-8100-ec69e635c8ed admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.754810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e3ce81-fd60-488b-acd4-616de60ed1ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.755419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e3ce81-fd60-488b-acd4-616de60ed1ea admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.755690 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e3ce81-fd60-488b-acd4-616de60ed1ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.755964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e3ce81-fd60-488b-acd4-616de60ed1ea admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.756626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e3ce81-fd60-488b-acd4-616de60ed1ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.758538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e3ce81-fd60-488b-acd4-616de60ed1ea admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.779743 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1295/2574] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:22.780871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1280/2575] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:22 2022] POST /identity/v3/users => generated 283 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:43:22.788234 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372c36ac-d58a-4582-a5bf-3f7d37353254 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.789018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3259d5-4f87-4a83-8f1d-bc897b6a08a5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.789307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3259d5-4f87-4a83-8f1d-bc897b6a08a5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.789622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-372c36ac-d58a-4582-a5bf-3f7d37353254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:22.789771 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3259d5-4f87-4a83-8f1d-bc897b6a08a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.802568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372c36ac-d58a-4582-a5bf-3f7d37353254 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.807748 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da7b5e-0098-4ac0-a89f-dbedd1192689 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.808371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da7b5e-0098-4ac0-a89f-dbedd1192689 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.808658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da7b5e-0098-4ac0-a89f-dbedd1192689 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.808940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da7b5e-0098-4ac0-a89f-dbedd1192689 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.809580 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32da7b5e-0098-4ac0-a89f-dbedd1192689 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.811202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32da7b5e-0098-4ac0-a89f-dbedd1192689 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.811666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1f3259d5-4f87-4a83-8f1d-bc897b6a08a5 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:22.812951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1f3259d5-4f87-4a83-8f1d-bc897b6a08a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:22.821202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1281/2576] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:22.824662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1296/2577] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:22 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:22.825369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3259d5-4f87-4a83-8f1d-bc897b6a08a5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.825783 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da7b5e-0098-4ac0-a89f-dbedd1192689 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.836241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3259d5-4f87-4a83-8f1d-bc897b6a08a5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.838224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da7b5e-0098-4ac0-a89f-dbedd1192689 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.842796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84583aa-aacb-411a-b71d-8995fd36ef58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.843401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84583aa-aacb-411a-b71d-8995fd36ef58 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.843570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c619e4-8ed1-443d-925b-c889dcdee670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.843715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84583aa-aacb-411a-b71d-8995fd36ef58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.843937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84583aa-aacb-411a-b71d-8995fd36ef58 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.844164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c619e4-8ed1-443d-925b-c889dcdee670 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.844982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c619e4-8ed1-443d-925b-c889dcdee670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.845173 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84583aa-aacb-411a-b71d-8995fd36ef58 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:22.845365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c619e4-8ed1-443d-925b-c889dcdee670 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.845491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84583aa-aacb-411a-b71d-8995fd36ef58 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.845911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c619e4-8ed1-443d-925b-c889dcdee670 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.847350 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84583aa-aacb-411a-b71d-8995fd36ef58 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.847522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c619e4-8ed1-443d-925b-c889dcdee670 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.857776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1297/2578] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:22.862485 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c619e4-8ed1-443d-925b-c889dcdee670 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.869092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1282/2579] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:43:22.870361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84583aa-aacb-411a-b71d-8995fd36ef58 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.872372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c619e4-8ed1-443d-925b-c889dcdee670 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.877643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c295bd7-0a96-431c-be96-483fa51a6821 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.878279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c295bd7-0a96-431c-be96-483fa51a6821 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.881369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84583aa-aacb-411a-b71d-8995fd36ef58 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.884609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c295bd7-0a96-431c-be96-483fa51a6821 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.884922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c295bd7-0a96-431c-be96-483fa51a6821 admin admin] PATH_INFO: `/v3/projects/716ebf15d84d47a0b50aca500cf3ea22/users/95a3a8a71266499fb9cabd183c874114/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.886095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1408734-1347-402d-958b-63bbcdf93d54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.886644 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1408734-1347-402d-958b-63bbcdf93d54 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.886908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1408734-1347-402d-958b-63bbcdf93d54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.887184 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1408734-1347-402d-958b-63bbcdf93d54 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.887803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1408734-1347-402d-958b-63bbcdf93d54 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.889808 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1408734-1347-402d-958b-63bbcdf93d54 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.902213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c295bd7-0a96-431c-be96-483fa51a6821 admin admin] RBAC: Authorizing `identity:create_grant(project_id=716ebf15d84d47a0b50aca500cf3ea22, user_id=95a3a8a71266499fb9cabd183c874114, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:22.905330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c295bd7-0a96-431c-be96-483fa51a6821 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:22.918783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1283/2580] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:22 2022] POST /identity/v3/projects => generated 365 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:22.919161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1298/2581] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:22 2022] PUT /identity/v3/projects/716ebf15d84d47a0b50aca500cf3ea22/users/95a3a8a71266499fb9cabd183c874114/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:22.925510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1408734-1347-402d-958b-63bbcdf93d54 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:22.930698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c0ca8-2482-40c2-a80b-732b5b57b4f6 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.930993 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c0ca8-2482-40c2-a80b-732b5b57b4f6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.931272 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c0ca8-2482-40c2-a80b-732b5b57b4f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.960631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-c53c0ca8-2482-40c2-a80b-732b5b57b4f6 None None] MFA Rules not processed for user `95a3a8a71266499fb9cabd183c874114`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:22.961863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c53c0ca8-2482-40c2-a80b-732b5b57b4f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:22.963157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:22.991984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1408734-1347-402d-958b-63bbcdf93d54 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:22.996738 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5612979-f614-4bb3-ac59-47134d8d8fb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:22.997339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5612979-f614-4bb3-ac59-47134d8d8fb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:22.997622 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5612979-f614-4bb3-ac59-47134d8d8fb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:22.997893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5612979-f614-4bb3-ac59-47134d8d8fb0 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:22.998544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5612979-f614-4bb3-ac59-47134d8d8fb0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.000504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5612979-f614-4bb3-ac59-47134d8d8fb0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.023572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1284/2582] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:22 2022] POST /identity/v3/users => generated 283 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:43:23.027607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5612979-f614-4bb3-ac59-47134d8d8fb0 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.029137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1299/2583] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:22 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:23.033766 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c0ca8-2482-40c2-a80b-732b5b57b4f6 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.038167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5612979-f614-4bb3-ac59-47134d8d8fb0 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.042853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a44ec5-ed9a-4b65-b401-ce55576b9867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.043454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a44ec5-ed9a-4b65-b401-ce55576b9867 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.043737 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a44ec5-ed9a-4b65-b401-ce55576b9867 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.044010 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a44ec5-ed9a-4b65-b401-ce55576b9867 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.044696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a44ec5-ed9a-4b65-b401-ce55576b9867 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.045936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c0ca8-2482-40c2-a80b-732b5b57b4f6 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.046305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a44ec5-ed9a-4b65-b401-ce55576b9867 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.051286 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0493a8-54c0-4a24-a3b9-9ec98c4c0bdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.051914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0493a8-54c0-4a24-a3b9-9ec98c4c0bdd admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.052196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0493a8-54c0-4a24-a3b9-9ec98c4c0bdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.052506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0493a8-54c0-4a24-a3b9-9ec98c4c0bdd admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.053136 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0493a8-54c0-4a24-a3b9-9ec98c4c0bdd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.055018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0493a8-54c0-4a24-a3b9-9ec98c4c0bdd admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.057567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1285/2584] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:23.061486 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a44ec5-ed9a-4b65-b401-ce55576b9867 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.073533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a44ec5-ed9a-4b65-b401-ce55576b9867 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.079211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16975bb-774b-4fcc-b801-2e3bc413d704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.079813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16975bb-774b-4fcc-b801-2e3bc413d704 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.080099 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16975bb-774b-4fcc-b801-2e3bc413d704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.080684 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16975bb-774b-4fcc-b801-2e3bc413d704 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.081344 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16975bb-774b-4fcc-b801-2e3bc413d704 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.083108 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16975bb-774b-4fcc-b801-2e3bc413d704 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.087572 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1300/2585] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:23 2022] POST /identity/v3/projects => generated 363 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:43:23.088670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0493a8-54c0-4a24-a3b9-9ec98c4c0bdd admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.098653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1286/2586] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:23 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:23.099680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16975bb-774b-4fcc-b801-2e3bc413d704 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.110772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16975bb-774b-4fcc-b801-2e3bc413d704 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.111373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:DELETE=1,keystone:SELECT=97,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:23.159268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0493a8-54c0-4a24-a3b9-9ec98c4c0bdd admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.163996 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b52b2d-d0c1-49f6-b064-0d80449a11b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.164629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b52b2d-d0c1-49f6-b064-0d80449a11b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.164906 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b52b2d-d0c1-49f6-b064-0d80449a11b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.165180 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b52b2d-d0c1-49f6-b064-0d80449a11b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.165831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b52b2d-d0c1-49f6-b064-0d80449a11b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.167788 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b52b2d-d0c1-49f6-b064-0d80449a11b3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.167968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1e14fe-8f76-4625-94c2-7deda2c52e27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.168643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1e14fe-8f76-4625-94c2-7deda2c52e27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.168925 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1e14fe-8f76-4625-94c2-7deda2c52e27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.169202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1e14fe-8f76-4625-94c2-7deda2c52e27 admin admin] PATH_INFO: `/v3/projects/2e13943779164d999b6421d5a893ab65/users/2cd25c080a6b4e4db7cc093e8e850b4a/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.185339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1e14fe-8f76-4625-94c2-7deda2c52e27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e13943779164d999b6421d5a893ab65, user_id=2cd25c080a6b4e4db7cc093e8e850b4a, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.187991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1e14fe-8f76-4625-94c2-7deda2c52e27 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.191083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1301/2587] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:23 2022] POST /identity/v3/users => generated 288 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:23.195569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b52b2d-d0c1-49f6-b064-0d80449a11b3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.202562 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1287/2588] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:23 2022] PUT /identity/v3/projects/2e13943779164d999b6421d5a893ab65/users/2cd25c080a6b4e4db7cc093e8e850b4a/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:23.205610 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b52b2d-d0c1-49f6-b064-0d80449a11b3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.213057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6404761b-596c-499e-b2ac-9b54e9dc0284 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.213372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6404761b-596c-499e-b2ac-9b54e9dc0284 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.213647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6404761b-596c-499e-b2ac-9b54e9dc0284 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.242326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6404761b-596c-499e-b2ac-9b54e9dc0284 None None] MFA Rules not processed for user `2cd25c080a6b4e4db7cc093e8e850b4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:23.243545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6404761b-596c-499e-b2ac-9b54e9dc0284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:23.263680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb41f663-b832-4f73-bce9-8842f77f8503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.264309 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb41f663-b832-4f73-bce9-8842f77f8503 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.264627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb41f663-b832-4f73-bce9-8842f77f8503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.264897 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb41f663-b832-4f73-bce9-8842f77f8503 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.265562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb41f663-b832-4f73-bce9-8842f77f8503 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.268063 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb41f663-b832-4f73-bce9-8842f77f8503 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.281838 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1302/2589] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:23 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:23.283565 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb41f663-b832-4f73-bce9-8842f77f8503 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.293318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb41f663-b832-4f73-bce9-8842f77f8503 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.297990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc74ce7-f6c5-440d-970d-27ea6d191574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.298629 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc74ce7-f6c5-440d-970d-27ea6d191574 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.298902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc74ce7-f6c5-440d-970d-27ea6d191574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.299176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc74ce7-f6c5-440d-970d-27ea6d191574 admin admin] PATH_INFO: `/v3/projects/e3736526f0ac47ea9747f6284ac1df6f/users/e7aee1fa72f8485e9a269e9e97c8356d/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.310472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1288/2590] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:23 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:23.317365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc74ce7-f6c5-440d-970d-27ea6d191574 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e3736526f0ac47ea9747f6284ac1df6f, user_id=e7aee1fa72f8485e9a269e9e97c8356d, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.319978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc74ce7-f6c5-440d-970d-27ea6d191574 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.322596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6404761b-596c-499e-b2ac-9b54e9dc0284 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.323983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6404761b-596c-499e-b2ac-9b54e9dc0284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:23.333846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1303/2591] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:23 2022] PUT /identity/v3/projects/e3736526f0ac47ea9747f6284ac1df6f/users/e7aee1fa72f8485e9a269e9e97c8356d/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:23.337647 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc74ce7-f6c5-440d-970d-27ea6d191574 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.343346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:23.388243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6404761b-596c-499e-b2ac-9b54e9dc0284 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.400417 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc74ce7-f6c5-440d-970d-27ea6d191574 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.405180 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846cd747-6b1e-43f0-9b86-13d3d16c7ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.405776 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846cd747-6b1e-43f0-9b86-13d3d16c7ad7 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.406049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846cd747-6b1e-43f0-9b86-13d3d16c7ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.406320 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846cd747-6b1e-43f0-9b86-13d3d16c7ad7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.407012 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846cd747-6b1e-43f0-9b86-13d3d16c7ad7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.408946 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846cd747-6b1e-43f0-9b86-13d3d16c7ad7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.418584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1304/2592] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:23 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:23.422801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846cd747-6b1e-43f0-9b86-13d3d16c7ad7 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.432564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846cd747-6b1e-43f0-9b86-13d3d16c7ad7 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.437243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0087af9-ac8f-4acf-87ea-e05e9b3dd3cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.437879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0087af9-ac8f-4acf-87ea-e05e9b3dd3cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.438690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0087af9-ac8f-4acf-87ea-e05e9b3dd3cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.439018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0087af9-ac8f-4acf-87ea-e05e9b3dd3cc admin admin] PATH_INFO: `/v3/projects/e3736526f0ac47ea9747f6284ac1df6f/users/e7aee1fa72f8485e9a269e9e97c8356d/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.441254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0087af9-ac8f-4acf-87ea-e05e9b3dd3cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=e3736526f0ac47ea9747f6284ac1df6f, user_id=e7aee1fa72f8485e9a269e9e97c8356d, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.441377 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81732173-e927-4c41-83b5-96ceda6cd507 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.441983 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81732173-e927-4c41-83b5-96ceda6cd507 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.442263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81732173-e927-4c41-83b5-96ceda6cd507 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.442534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81732173-e927-4c41-83b5-96ceda6cd507 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.443891 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0087af9-ac8f-4acf-87ea-e05e9b3dd3cc admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.450017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:23.455419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1305/2593] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:23 2022] PUT /identity/v3/projects/e3736526f0ac47ea9747f6284ac1df6f/users/e7aee1fa72f8485e9a269e9e97c8356d/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:23.465926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2513d74-69fc-4495-b075-713971a6d05b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.466212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2513d74-69fc-4495-b075-713971a6d05b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.466486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2513d74-69fc-4495-b075-713971a6d05b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.494901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-a2513d74-69fc-4495-b075-713971a6d05b None None] MFA Rules not processed for user `e7aee1fa72f8485e9a269e9e97c8356d`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:23.496123 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-a2513d74-69fc-4495-b075-713971a6d05b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:23.504525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81732173-e927-4c41-83b5-96ceda6cd507 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.506175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81732173-e927-4c41-83b5-96ceda6cd507 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.515156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1289/2594] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:23 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:43:23.567683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1306/2595] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:23 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:23.579766 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81732173-e927-4c41-83b5-96ceda6cd507 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.581255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-81732173-e927-4c41-83b5-96ceda6cd507 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:23.593331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81732173-e927-4c41-83b5-96ceda6cd507 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.644111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfea2a7-8b05-491d-a64f-5d65b9344d6e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.644827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfea2a7-8b05-491d-a64f-5d65b9344d6e service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.645159 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfea2a7-8b05-491d-a64f-5d65b9344d6e service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.645495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfea2a7-8b05-491d-a64f-5d65b9344d6e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.654851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfea2a7-8b05-491d-a64f-5d65b9344d6e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.656449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfea2a7-8b05-491d-a64f-5d65b9344d6e service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.667284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1290/2596] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:23 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:23.749963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2513d74-69fc-4495-b075-713971a6d05b None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:23.760295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2513d74-69fc-4495-b075-713971a6d05b None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:23.768447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236dd3a9-d5c6-4c0a-9cd6-202f3d51a4a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:23.768447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236dd3a9-d5c6-4c0a-9cd6-202f3d51a4a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:23.768447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236dd3a9-d5c6-4c0a-9cd6-202f3d51a4a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:23.768447 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236dd3a9-d5c6-4c0a-9cd6-202f3d51a4a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:23.840983 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236dd3a9-d5c6-4c0a-9cd6-202f3d51a4a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:23.842422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236dd3a9-d5c6-4c0a-9cd6-202f3d51a4a5 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:23.852036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1307/2597] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:23 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:26.406011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcbb047-6aef-4c46-8129-5b8ac9b9cacb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:26.406480 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcbb047-6aef-4c46-8129-5b8ac9b9cacb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:26.406575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcbb047-6aef-4c46-8129-5b8ac9b9cacb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:26.427715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-abcbb047-6aef-4c46-8129-5b8ac9b9cacb None None] MFA Rules not processed for user `64c98c1da1a847389fa035d86f6326de`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:26.428950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-abcbb047-6aef-4c46-8129-5b8ac9b9cacb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:26.449431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:26.486203 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1291/2598] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:26 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:26.743816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236dd3a9-d5c6-4c0a-9cd6-202f3d51a4a5 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:26.823563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236dd3a9-d5c6-4c0a-9cd6-202f3d51a4a5 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:26.828383 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:26.830043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:26.830323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:26.830596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] PATH_INFO: `/v3/users/95a3a8a71266499fb9cabd183c874114` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:26.831736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=95a3a8a71266499fb9cabd183c874114)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:26.833686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:26.840669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcbb047-6aef-4c46-8129-5b8ac9b9cacb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:26.842069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-abcbb047-6aef-4c46-8129-5b8ac9b9cacb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:26.864421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:26.885526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95a3a8a71266499fb9cabd183c874114'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:26.893945 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95a3a8a71266499fb9cabd183c874114'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:26.894613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95a3a8a71266499fb9cabd183c874114'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:26.895293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95a3a8a71266499fb9cabd183c874114'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:26.906803 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcbb047-6aef-4c46-8129-5b8ac9b9cacb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:26.910064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2a685e80-4858-4dde-9fb5-1e822c2868e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95a3a8a71266499fb9cabd183c874114'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:26.910543 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a537bd9-603e-41a1-8332-8d6cdf9edd40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:26.911148 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a537bd9-603e-41a1-8332-8d6cdf9edd40 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:26.911627 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a537bd9-603e-41a1-8332-8d6cdf9edd40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:26.912121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a537bd9-603e-41a1-8332-8d6cdf9edd40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:26.913758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8a537bd9-603e-41a1-8332-8d6cdf9edd40 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:26.923069 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1308/2599] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:26 2022] DELETE /identity/v3/users/95a3a8a71266499fb9cabd183c874114 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:26.924289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a537bd9-603e-41a1-8332-8d6cdf9edd40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:26.925751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a537bd9-603e-41a1-8332-8d6cdf9edd40 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:26.927510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec9a452-faa6-4b0a-ae86-8b2e8d046444 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:26.927801 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec9a452-faa6-4b0a-ae86-8b2e8d046444 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:26.928078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec9a452-faa6-4b0a-ae86-8b2e8d046444 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:26.934204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1292/2600] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:26 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:26.949933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-8ec9a452-faa6-4b0a-ae86-8b2e8d046444 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:26.951322 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8ec9a452-faa6-4b0a-ae86-8b2e8d046444 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:27.015858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1309/2601] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:27.025899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a537bd9-603e-41a1-8332-8d6cdf9edd40 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:27.027666 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8a537bd9-603e-41a1-8332-8d6cdf9edd40 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:27.045794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a537bd9-603e-41a1-8332-8d6cdf9edd40 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:27.108963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef78949-8d7b-453d-b427-15bfdd8548e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:27.109574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef78949-8d7b-453d-b427-15bfdd8548e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:27.110512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef78949-8d7b-453d-b427-15bfdd8548e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:27.110798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef78949-8d7b-453d-b427-15bfdd8548e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:27.121362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef78949-8d7b-453d-b427-15bfdd8548e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:27.122822 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef78949-8d7b-453d-b427-15bfdd8548e3 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:27.134039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1293/2602] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:27.417023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec9a452-faa6-4b0a-ae86-8b2e8d046444 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:27.419067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8ec9a452-faa6-4b0a-ae86-8b2e8d046444 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:27.445037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec9a452-faa6-4b0a-ae86-8b2e8d046444 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:27.451051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:27.451998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:27.452400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:27.452760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] PATH_INFO: `/v3/projects/716ebf15d84d47a0b50aca500cf3ea22` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:27.454774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] RBAC: Authorizing `identity:delete_project(project_id=716ebf15d84d47a0b50aca500cf3ea22)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:27.456747 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:27.518468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '716ebf15d84d47a0b50aca500cf3ea22', 'request_id': 'req-4285d70e-2655-44d5-b945-3ec44357b591'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:27.526380 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '716ebf15d84d47a0b50aca500cf3ea22', 'request_id': 'req-4285d70e-2655-44d5-b945-3ec44357b591'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:27.533815 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '716ebf15d84d47a0b50aca500cf3ea22', 'request_id': 'req-4285d70e-2655-44d5-b945-3ec44357b591'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:27.551964 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] The token cache is being invalidate because project 716ebf15d84d47a0b50aca500cf3ea22 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:27.552564 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:27.553426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:27.555238 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1310/2603] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:27 2022] DELETE /identity/v3/projects/716ebf15d84d47a0b50aca500cf3ea22 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:27.559267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef78949-8d7b-453d-b427-15bfdd8548e3 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:27.569968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef78949-8d7b-453d-b427-15bfdd8548e3 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:27.633797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:27.634423 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:27.634703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:27.635037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] PATH_INFO: `/v3/users/e7aee1fa72f8485e9a269e9e97c8356d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:27.636144 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7aee1fa72f8485e9a269e9e97c8356d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:27.638132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:27.659919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:27.687435 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7aee1fa72f8485e9a269e9e97c8356d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:27.695823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7aee1fa72f8485e9a269e9e97c8356d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:27.696491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7aee1fa72f8485e9a269e9e97c8356d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:27.697110 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7aee1fa72f8485e9a269e9e97c8356d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:27.705867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-473260d3-02d2-4c6b-b89b-b10fe05fedaa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7aee1fa72f8485e9a269e9e97c8356d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:27.724858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1294/2604] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:27 2022] DELETE /identity/v3/users/e7aee1fa72f8485e9a269e9e97c8356d => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:28.027348 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:28.029092 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:28.052416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285d70e-2655-44d5-b945-3ec44357b591 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:28.085275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:28.123534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:28.124250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.124625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.124969 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] PATH_INFO: `/v3/projects/e3736526f0ac47ea9747f6284ac1df6f` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.127002 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e3736526f0ac47ea9747f6284ac1df6f)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:28.129146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:28.184563 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e3736526f0ac47ea9747f6284ac1df6f', 'request_id': 'req-2dc63320-181b-4277-9b9a-b8de2d3e2287'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:28.192832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e3736526f0ac47ea9747f6284ac1df6f', 'request_id': 'req-2dc63320-181b-4277-9b9a-b8de2d3e2287'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:28.198916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e3736526f0ac47ea9747f6284ac1df6f', 'request_id': 'req-2dc63320-181b-4277-9b9a-b8de2d3e2287'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:28.214741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] The token cache is being invalidate because project e3736526f0ac47ea9747f6284ac1df6f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:28.215027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:28.215674 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-2dc63320-181b-4277-9b9a-b8de2d3e2287 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:28.217184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1311/2605] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:28 2022] DELETE /identity/v3/projects/e3736526f0ac47ea9747f6284ac1df6f => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:28.224275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6a9267-a988-4ea6-913e-d7b466ff1a88 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.224709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6a9267-a988-4ea6-913e-d7b466ff1a88 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.224857 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6a9267-a988-4ea6-913e-d7b466ff1a88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.249827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-cc6a9267-a988-4ea6-913e-d7b466ff1a88 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:28.249827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cc6a9267-a988-4ea6-913e-d7b466ff1a88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:28.317304 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1295/2606] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:28.325103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd1ba75-18fa-4471-8fb8-3dba7c037b17 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.325415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd1ba75-18fa-4471-8fb8-3dba7c037b17 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.325688 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd1ba75-18fa-4471-8fb8-3dba7c037b17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.348162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-9cd1ba75-18fa-4471-8fb8-3dba7c037b17 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:28.356178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9cd1ba75-18fa-4471-8fb8-3dba7c037b17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:28.365528 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1312/2607] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:28.369786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6a9267-a988-4ea6-913e-d7b466ff1a88 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:28.379919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6a9267-a988-4ea6-913e-d7b466ff1a88 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:28.384525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4b06d9-6c55-4fde-b018-e1eeedc98ccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:28.385128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4b06d9-6c55-4fde-b018-e1eeedc98ccd admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.385415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4b06d9-6c55-4fde-b018-e1eeedc98ccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.385686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4b06d9-6c55-4fde-b018-e1eeedc98ccd admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.386358 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4b06d9-6c55-4fde-b018-e1eeedc98ccd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:28.386663 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4b06d9-6c55-4fde-b018-e1eeedc98ccd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:28.388247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4b06d9-6c55-4fde-b018-e1eeedc98ccd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:28.400586 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1296/2608] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:43:28.404930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd1ba75-18fa-4471-8fb8-3dba7c037b17 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:28.415957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd1ba75-18fa-4471-8fb8-3dba7c037b17 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:28.420621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4c2008-f499-45b2-b7c3-329dab529b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:28.421212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4c2008-f499-45b2-b7c3-329dab529b3a admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.421710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4c2008-f499-45b2-b7c3-329dab529b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.421837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4c2008-f499-45b2-b7c3-329dab529b3a admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.423053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4c2008-f499-45b2-b7c3-329dab529b3a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:28.425813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4c2008-f499-45b2-b7c3-329dab529b3a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:28.456825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1313/2609] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:28 2022] POST /identity/v3/projects => generated 381 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:28.458197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4b06d9-6c55-4fde-b018-e1eeedc98ccd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:28.468401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4b06d9-6c55-4fde-b018-e1eeedc98ccd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:28.526874 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0041d1-adb1-4064-bd93-c425649f82b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:28.527547 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0041d1-adb1-4064-bd93-c425649f82b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.527898 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0041d1-adb1-4064-bd93-c425649f82b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.528241 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0041d1-adb1-4064-bd93-c425649f82b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.528910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0041d1-adb1-4064-bd93-c425649f82b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:28.530814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0041d1-adb1-4064-bd93-c425649f82b3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:28.552976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1297/2610] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:28 2022] POST /identity/v3/users => generated 291 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:28.556932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4c2008-f499-45b2-b7c3-329dab529b3a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:28.567933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4c2008-f499-45b2-b7c3-329dab529b3a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:28.572714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4bb2ed-6f52-4448-9307-fe8d3a90bd86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:28.573318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4bb2ed-6f52-4448-9307-fe8d3a90bd86 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.573602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4bb2ed-6f52-4448-9307-fe8d3a90bd86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.573962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4bb2ed-6f52-4448-9307-fe8d3a90bd86 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.574886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4bb2ed-6f52-4448-9307-fe8d3a90bd86 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:28.576515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4bb2ed-6f52-4448-9307-fe8d3a90bd86 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:28.590133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1314/2611] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:28.595107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0041d1-adb1-4064-bd93-c425649f82b3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:28.612491 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0041d1-adb1-4064-bd93-c425649f82b3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:28.617319 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48de84d-1051-460b-be27-daa757e8095d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:28.618068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48de84d-1051-460b-be27-daa757e8095d admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.618411 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48de84d-1051-460b-be27-daa757e8095d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.618750 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48de84d-1051-460b-be27-daa757e8095d admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.619591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48de84d-1051-460b-be27-daa757e8095d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:28.621565 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48de84d-1051-460b-be27-daa757e8095d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:28.636482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1298/2612] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:28 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:28.640618 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4bb2ed-6f52-4448-9307-fe8d3a90bd86 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:28.651785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4bb2ed-6f52-4448-9307-fe8d3a90bd86 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:28.657328 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5897e10-da95-4ece-9813-d89737bdd779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:28.657817 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5897e10-da95-4ece-9813-d89737bdd779 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.658161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5897e10-da95-4ece-9813-d89737bdd779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.658591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5897e10-da95-4ece-9813-d89737bdd779 admin admin] PATH_INFO: `/v3/projects/5301fac31fe840ff9e5d98f483611823/users/a740cb910cf54e789ecd921d73c15e46/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.677907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5897e10-da95-4ece-9813-d89737bdd779 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5301fac31fe840ff9e5d98f483611823, user_id=a740cb910cf54e789ecd921d73c15e46, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:28.680942 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5897e10-da95-4ece-9813-d89737bdd779 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:28.712553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1315/2613] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:28 2022] PUT /identity/v3/projects/5301fac31fe840ff9e5d98f483611823/users/a740cb910cf54e789ecd921d73c15e46/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:28.712709 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8dbec3-92f4-4c7a-b9ed-c55fd7fc497b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.713038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8dbec3-92f4-4c7a-b9ed-c55fd7fc497b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.713321 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8dbec3-92f4-4c7a-b9ed-c55fd7fc497b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.722818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:28.751978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0f8dbec3-92f4-4c7a-b9ed-c55fd7fc497b None None] MFA Rules not processed for user `a740cb910cf54e789ecd921d73c15e46`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:28.753815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0f8dbec3-92f4-4c7a-b9ed-c55fd7fc497b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:28.850643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1299/2614] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:28 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:28.855620 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5897e10-da95-4ece-9813-d89737bdd779 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:28.931106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:28.949557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5897e10-da95-4ece-9813-d89737bdd779 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:28.958071 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3330f5-265a-4a38-9884-4a09b0b99dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:28.958762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3330f5-265a-4a38-9884-4a09b0b99dc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:28.959109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3330f5-265a-4a38-9884-4a09b0b99dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:28.959648 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3330f5-265a-4a38-9884-4a09b0b99dc9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:28.960370 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3330f5-265a-4a38-9884-4a09b0b99dc9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:28.962335 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3330f5-265a-4a38-9884-4a09b0b99dc9 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:28.993077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1316/2615] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:28 2022] POST /identity/v3/projects => generated 383 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:28.996981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8dbec3-92f4-4c7a-b9ed-c55fd7fc497b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:29.008805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8dbec3-92f4-4c7a-b9ed-c55fd7fc497b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:29.071509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dc6889-cb35-42cd-a080-81a19db0adfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:29.072126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dc6889-cb35-42cd-a080-81a19db0adfe admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:29.072507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dc6889-cb35-42cd-a080-81a19db0adfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:29.072847 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dc6889-cb35-42cd-a080-81a19db0adfe admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:29.073565 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dc6889-cb35-42cd-a080-81a19db0adfe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:29.075563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dc6889-cb35-42cd-a080-81a19db0adfe admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:29.103667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1300/2616] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:28 2022] POST /identity/v3/users => generated 298 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:29.104939 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3330f5-265a-4a38-9884-4a09b0b99dc9 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:29.117394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3330f5-265a-4a38-9884-4a09b0b99dc9 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:29.124384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c7750e-3ac6-4e10-ad50-b05068ff8152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:29.125482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c7750e-3ac6-4e10-ad50-b05068ff8152 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:29.125886 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c7750e-3ac6-4e10-ad50-b05068ff8152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:29.126162 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c7750e-3ac6-4e10-ad50-b05068ff8152 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:29.126816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c7750e-3ac6-4e10-ad50-b05068ff8152 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:29.128640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c7750e-3ac6-4e10-ad50-b05068ff8152 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:29.139597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1317/2617] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:29.144772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dc6889-cb35-42cd-a080-81a19db0adfe admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:29.155008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dc6889-cb35-42cd-a080-81a19db0adfe admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:29.160630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faa92ae-7d0b-4b28-808c-f8b5dcd75711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:29.161291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faa92ae-7d0b-4b28-808c-f8b5dcd75711 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:29.161573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faa92ae-7d0b-4b28-808c-f8b5dcd75711 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:29.161851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faa92ae-7d0b-4b28-808c-f8b5dcd75711 admin admin] PATH_INFO: `/v3/projects/296ce2199a01493dad9a7cc024f092f7/users/bba8fc9a1c844c2db13ddfac8ab05077/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:29.177695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faa92ae-7d0b-4b28-808c-f8b5dcd75711 admin admin] RBAC: Authorizing `identity:create_grant(project_id=296ce2199a01493dad9a7cc024f092f7, user_id=bba8fc9a1c844c2db13ddfac8ab05077, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:29.180293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faa92ae-7d0b-4b28-808c-f8b5dcd75711 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:29.194544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1301/2618] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:29 2022] PUT /identity/v3/projects/296ce2199a01493dad9a7cc024f092f7/users/bba8fc9a1c844c2db13ddfac8ab05077/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:29.198512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c7750e-3ac6-4e10-ad50-b05068ff8152 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:29.209576 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c7750e-3ac6-4e10-ad50-b05068ff8152 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:29.272619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafcbcec-7057-475c-afa2-ac4b32136b62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:29.273226 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafcbcec-7057-475c-afa2-ac4b32136b62 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:29.273506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafcbcec-7057-475c-afa2-ac4b32136b62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:29.273774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafcbcec-7057-475c-afa2-ac4b32136b62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:29.274414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafcbcec-7057-475c-afa2-ac4b32136b62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:29.276062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafcbcec-7057-475c-afa2-ac4b32136b62 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:29.286237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1318/2619] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:29 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:29.290387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faa92ae-7d0b-4b28-808c-f8b5dcd75711 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:29.301976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faa92ae-7d0b-4b28-808c-f8b5dcd75711 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:29.306716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2141a-5062-4edc-8dc0-872956819f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:29.307498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2141a-5062-4edc-8dc0-872956819f31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:29.307777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2141a-5062-4edc-8dc0-872956819f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:29.308075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f2141a-5062-4edc-8dc0-872956819f31 admin admin] PATH_INFO: `/v3/projects/296ce2199a01493dad9a7cc024f092f7/users/bba8fc9a1c844c2db13ddfac8ab05077/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:29.310057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f2141a-5062-4edc-8dc0-872956819f31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=296ce2199a01493dad9a7cc024f092f7, user_id=bba8fc9a1c844c2db13ddfac8ab05077, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:29.312755 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f2141a-5062-4edc-8dc0-872956819f31 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:29.325005 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1302/2620] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:29 2022] PUT /identity/v3/projects/296ce2199a01493dad9a7cc024f092f7/users/bba8fc9a1c844c2db13ddfac8ab05077/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:29.335486 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963a269a-f728-4742-b640-f915431385c7 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:29.335773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963a269a-f728-4742-b640-f915431385c7 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:29.336048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963a269a-f728-4742-b640-f915431385c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:29.365992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-963a269a-f728-4742-b640-f915431385c7 None None] MFA Rules not processed for user `bba8fc9a1c844c2db13ddfac8ab05077`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:29.367320 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-963a269a-f728-4742-b640-f915431385c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:29.439997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1319/2621] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:29 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:29.451310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2141a-5062-4edc-8dc0-872956819f31 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:29.453284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b1f2141a-5062-4edc-8dc0-872956819f31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:29.520221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:29.534820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f2141a-5062-4edc-8dc0-872956819f31 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:29.539392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cc4a00-ef60-4cb5-8bba-a3af3ed2a80a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:29.540147 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cc4a00-ef60-4cb5-8bba-a3af3ed2a80a service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:29.540527 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cc4a00-ef60-4cb5-8bba-a3af3ed2a80a service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:29.540768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cc4a00-ef60-4cb5-8bba-a3af3ed2a80a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:29.551166 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cc4a00-ef60-4cb5-8bba-a3af3ed2a80a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:29.552674 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cc4a00-ef60-4cb5-8bba-a3af3ed2a80a service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:29.565910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1303/2622] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:29 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:29.678440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963a269a-f728-4742-b640-f915431385c7 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:29.688937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963a269a-f728-4742-b640-f915431385c7 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:29.693475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5ff68e-2d5e-4afd-a3c2-40f3b3e78265 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:29.694323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5ff68e-2d5e-4afd-a3c2-40f3b3e78265 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:29.694787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5ff68e-2d5e-4afd-a3c2-40f3b3e78265 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:29.695224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5ff68e-2d5e-4afd-a3c2-40f3b3e78265 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:29.755678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:29.773022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5ff68e-2d5e-4afd-a3c2-40f3b3e78265 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:29.774473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5ff68e-2d5e-4afd-a3c2-40f3b3e78265 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:29.785028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1320/2623] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:29 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:32.345702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cc4a00-ef60-4cb5-8bba-a3af3ed2a80a service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:32.347051 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-67cc4a00-ef60-4cb5-8bba-a3af3ed2a80a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:32.412225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cc4a00-ef60-4cb5-8bba-a3af3ed2a80a service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:32.418376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:32.419557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:32.419923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5ff68e-2d5e-4afd-a3c2-40f3b3e78265 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:32.420399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:32.420896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] PATH_INFO: `/v3/users/2cd25c080a6b4e4db7cc093e8e850b4a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:32.422910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2cd25c080a6b4e4db7cc093e8e850b4a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:32.425028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:32.431091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5ff68e-2d5e-4afd-a3c2-40f3b3e78265 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:32.436427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:32.437060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:32.437343 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:32.438174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] PATH_INFO: `/v3/users/a740cb910cf54e789ecd921d73c15e46` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:32.439578 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a740cb910cf54e789ecd921d73c15e46)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:32.441570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:32.476495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2cd25c080a6b4e4db7cc093e8e850b4a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:32.484941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2cd25c080a6b4e4db7cc093e8e850b4a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:32.485664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2cd25c080a6b4e4db7cc093e8e850b4a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:32.486347 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2cd25c080a6b4e4db7cc093e8e850b4a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:32.491154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a740cb910cf54e789ecd921d73c15e46'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:32.493748 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4f8a7039-43a1-44d6-be3a-42bdfc4fb84e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2cd25c080a6b4e4db7cc093e8e850b4a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:32.498418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a740cb910cf54e789ecd921d73c15e46'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:32.499111 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a740cb910cf54e789ecd921d73c15e46'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:32.499746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a740cb910cf54e789ecd921d73c15e46'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:32.512760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1304/2624] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:32 2022] DELETE /identity/v3/users/2cd25c080a6b4e4db7cc093e8e850b4a => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:32.513222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-81e63834-02f4-470b-b260-93c27b63aa3c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a740cb910cf54e789ecd921d73c15e46'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:32.517605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89893fe6-7002-43c6-890a-66c6128b6605 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:32.517890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89893fe6-7002-43c6-890a-66c6128b6605 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:32.518163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89893fe6-7002-43c6-890a-66c6128b6605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:32.528492 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1321/2625] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:32 2022] DELETE /identity/v3/users/a740cb910cf54e789ecd921d73c15e46 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:32.532878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed609af4-4a63-4991-ba97-b55be0159ca8 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:32.533166 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed609af4-4a63-4991-ba97-b55be0159ca8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:32.533440 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed609af4-4a63-4991-ba97-b55be0159ca8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:32.540876 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-89893fe6-7002-43c6-890a-66c6128b6605 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:32.542095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-89893fe6-7002-43c6-890a-66c6128b6605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:32.553739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-ed609af4-4a63-4991-ba97-b55be0159ca8 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:32.555316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ed609af4-4a63-4991-ba97-b55be0159ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:32.613912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1305/2626] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:32.622111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89893fe6-7002-43c6-890a-66c6128b6605 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:32.623721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-89893fe6-7002-43c6-890a-66c6128b6605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:32.627868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1322/2627] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:32.637479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed609af4-4a63-4991-ba97-b55be0159ca8 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:32.638104 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:32.639599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ed609af4-4a63-4991-ba97-b55be0159ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:32.699598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89893fe6-7002-43c6-890a-66c6128b6605 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:32.703940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2f9be8-3881-4a26-ad04-657e23d1cfdd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:32.704567 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2f9be8-3881-4a26-ad04-657e23d1cfdd service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:32.704851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2f9be8-3881-4a26-ad04-657e23d1cfdd service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:32.705124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2f9be8-3881-4a26-ad04-657e23d1cfdd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:32.708334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed609af4-4a63-4991-ba97-b55be0159ca8 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:32.713552 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26cd536-bf99-47df-a6f7-3039bd8f3205 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:32.714427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26cd536-bf99-47df-a6f7-3039bd8f3205 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:32.714827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26cd536-bf99-47df-a6f7-3039bd8f3205 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:32.715103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26cd536-bf99-47df-a6f7-3039bd8f3205 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:32.715455 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2f9be8-3881-4a26-ad04-657e23d1cfdd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:32.717039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2f9be8-3881-4a26-ad04-657e23d1cfdd service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:32.727260 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26cd536-bf99-47df-a6f7-3039bd8f3205 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:32.729314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26cd536-bf99-47df-a6f7-3039bd8f3205 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:32.729462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1306/2628] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:32.744543 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1323/2629] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:43:33.034550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2f9be8-3881-4a26-ad04-657e23d1cfdd service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.041538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8a2f9be8-3881-4a26-ad04-657e23d1cfdd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:33.042715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26cd536-bf99-47df-a6f7-3039bd8f3205 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.053608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e26cd536-bf99-47df-a6f7-3039bd8f3205 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:33.061735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2f9be8-3881-4a26-ad04-657e23d1cfdd service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.063281 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:33.071023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26cd536-bf99-47df-a6f7-3039bd8f3205 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.071615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.073371 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.073951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.074096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] PATH_INFO: `/v3/projects/2e13943779164d999b6421d5a893ab65` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.075485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e13943779164d999b6421d5a893ab65)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.080514 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.080701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.080701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.080701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.080701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] PATH_INFO: `/v3/projects/5301fac31fe840ff9e5d98f483611823` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.082230 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5301fac31fe840ff9e5d98f483611823)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.084732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.145830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5301fac31fe840ff9e5d98f483611823', 'request_id': 'req-5157d441-d355-45c2-87a5-7202dfa95807'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.149633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e13943779164d999b6421d5a893ab65', 'request_id': 'req-a372ea67-5a48-4e44-960d-519df2d2cd73'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.156888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5301fac31fe840ff9e5d98f483611823', 'request_id': 'req-5157d441-d355-45c2-87a5-7202dfa95807'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.158447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e13943779164d999b6421d5a893ab65', 'request_id': 'req-a372ea67-5a48-4e44-960d-519df2d2cd73'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.163109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5301fac31fe840ff9e5d98f483611823', 'request_id': 'req-5157d441-d355-45c2-87a5-7202dfa95807'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.169382 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e13943779164d999b6421d5a893ab65', 'request_id': 'req-a372ea67-5a48-4e44-960d-519df2d2cd73'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.181386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] The token cache is being invalidate because project 5301fac31fe840ff9e5d98f483611823 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:33.181763 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.182404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] The token cache is being invalidate because project 2e13943779164d999b6421d5a893ab65 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:33.182533 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.182696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.183528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a372ea67-5a48-4e44-960d-519df2d2cd73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.185268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1307/2630] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:33 2022] DELETE /identity/v3/projects/2e13943779164d999b6421d5a893ab65 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:33.188665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1324/2631] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:33 2022] DELETE /identity/v3/projects/5301fac31fe840ff9e5d98f483611823 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:33.190957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ed761b-33ea-41fd-8533-416fbaf7835d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.191249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ed761b-33ea-41fd-8533-416fbaf7835d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.191529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ed761b-33ea-41fd-8533-416fbaf7835d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.193150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.194809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:33.212791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-82ed761b-33ea-41fd-8533-416fbaf7835d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:33.214015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-82ed761b-33ea-41fd-8533-416fbaf7835d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:33.261774 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5157d441-d355-45c2-87a5-7202dfa95807 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.275724 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1308/2632] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:33.285528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e43e2a-0200-4584-9562-b04847d678a3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.285818 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e43e2a-0200-4584-9562-b04847d678a3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.286496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e43e2a-0200-4584-9562-b04847d678a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.307548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e7e43e2a-0200-4584-9562-b04847d678a3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:33.308927 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e7e43e2a-0200-4584-9562-b04847d678a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:33.318563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1309/2633] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:33.319463 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.320086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.320391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.320668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] PATH_INFO: `/v3/users/bba8fc9a1c844c2db13ddfac8ab05077` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.321800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bba8fc9a1c844c2db13ddfac8ab05077)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.322744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e43e2a-0200-4584-9562-b04847d678a3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.323705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.332660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e43e2a-0200-4584-9562-b04847d678a3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.337301 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10dce9e-018b-4016-8504-5021ba7d4d47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.337895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10dce9e-018b-4016-8504-5021ba7d4d47 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.338174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10dce9e-018b-4016-8504-5021ba7d4d47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.338444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10dce9e-018b-4016-8504-5021ba7d4d47 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.339083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10dce9e-018b-4016-8504-5021ba7d4d47 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:33.339388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10dce9e-018b-4016-8504-5021ba7d4d47 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.341053 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10dce9e-018b-4016-8504-5021ba7d4d47 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.354976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1310/2634] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:43:33.359131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10dce9e-018b-4016-8504-5021ba7d4d47 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.368107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:33.371130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10dce9e-018b-4016-8504-5021ba7d4d47 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.377836 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd128d7c-84fd-4296-a2dd-651443639f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.378738 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd128d7c-84fd-4296-a2dd-651443639f91 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.378925 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd128d7c-84fd-4296-a2dd-651443639f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.379312 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd128d7c-84fd-4296-a2dd-651443639f91 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.380041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd128d7c-84fd-4296-a2dd-651443639f91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.381378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bba8fc9a1c844c2db13ddfac8ab05077'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.382156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd128d7c-84fd-4296-a2dd-651443639f91 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.392750 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bba8fc9a1c844c2db13ddfac8ab05077'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.393890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bba8fc9a1c844c2db13ddfac8ab05077'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.394719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bba8fc9a1c844c2db13ddfac8ab05077'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.395190 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:33.411264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bba8fc9a1c844c2db13ddfac8ab05077'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.417125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1311/2635] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:33 2022] POST /identity/v3/projects => generated 363 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:33.418095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd128d7c-84fd-4296-a2dd-651443639f91 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.422433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bd128d7c-84fd-4296-a2dd-651443639f91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:33.428586 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1325/2636] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:33 2022] DELETE /identity/v3/users/bba8fc9a1c844c2db13ddfac8ab05077 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:33.501817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd128d7c-84fd-4296-a2dd-651443639f91 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.568067 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3714711a-2b61-453b-a074-52846caffd85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.568951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3714711a-2b61-453b-a074-52846caffd85 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.569231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3714711a-2b61-453b-a074-52846caffd85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.569504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3714711a-2b61-453b-a074-52846caffd85 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.570155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3714711a-2b61-453b-a074-52846caffd85 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.572101 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3714711a-2b61-453b-a074-52846caffd85 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.598230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1312/2637] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:33 2022] POST /identity/v3/users => generated 282 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:33.602028 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.614979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f16ecfc-afb7-4f2a-ac6c-d0aa6b818212 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.619541 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733514de-fb2e-4405-82f6-c3720e5a23d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.620137 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733514de-fb2e-4405-82f6-c3720e5a23d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.620436 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733514de-fb2e-4405-82f6-c3720e5a23d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.620707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733514de-fb2e-4405-82f6-c3720e5a23d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.621347 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733514de-fb2e-4405-82f6-c3720e5a23d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.622925 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733514de-fb2e-4405-82f6-c3720e5a23d4 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.634003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1326/2638] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:33.638015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3714711a-2b61-453b-a074-52846caffd85 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.649864 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3714711a-2b61-453b-a074-52846caffd85 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.655136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed8c884-0b85-44aa-829d-a8b756c0102a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.655787 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed8c884-0b85-44aa-829d-a8b756c0102a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.656089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed8c884-0b85-44aa-829d-a8b756c0102a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.656422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed8c884-0b85-44aa-829d-a8b756c0102a admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.657072 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed8c884-0b85-44aa-829d-a8b756c0102a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.658675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed8c884-0b85-44aa-829d-a8b756c0102a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.669201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1313/2639] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:33.673888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733514de-fb2e-4405-82f6-c3720e5a23d4 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.683894 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733514de-fb2e-4405-82f6-c3720e5a23d4 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.688643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2035172-e3a3-4a99-8d5e-c4e49d171124 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.689555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2035172-e3a3-4a99-8d5e-c4e49d171124 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.689837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2035172-e3a3-4a99-8d5e-c4e49d171124 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.690114 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2035172-e3a3-4a99-8d5e-c4e49d171124 admin admin] PATH_INFO: `/v3/projects/25729b643dd943e482c777e4a6f2c9d6/users/8c517dedb30841a38e6139cdb9540e12/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.707023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2035172-e3a3-4a99-8d5e-c4e49d171124 admin admin] RBAC: Authorizing `identity:create_grant(project_id=25729b643dd943e482c777e4a6f2c9d6, user_id=8c517dedb30841a38e6139cdb9540e12, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.710102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2035172-e3a3-4a99-8d5e-c4e49d171124 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.722871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed8c884-0b85-44aa-829d-a8b756c0102a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.724537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1327/2640] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:33 2022] PUT /identity/v3/projects/25729b643dd943e482c777e4a6f2c9d6/users/8c517dedb30841a38e6139cdb9540e12/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:33.724692 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9ed8c884-0b85-44aa-829d-a8b756c0102a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:33.734101 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37328e9e-e0ab-4047-aa7a-c5dce2c41928 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.735360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37328e9e-e0ab-4047-aa7a-c5dce2c41928 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.735741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37328e9e-e0ab-4047-aa7a-c5dce2c41928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.739758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed8c884-0b85-44aa-829d-a8b756c0102a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.767785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-37328e9e-e0ab-4047-aa7a-c5dce2c41928 None None] MFA Rules not processed for user `8c517dedb30841a38e6139cdb9540e12`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:33.769076 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-37328e9e-e0ab-4047-aa7a-c5dce2c41928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:33.846171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1328/2641] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:33 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:33.850878 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37328e9e-e0ab-4047-aa7a-c5dce2c41928 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.916709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37328e9e-e0ab-4047-aa7a-c5dce2c41928 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:33.921439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c8457c-f604-439a-87e6-2f602da906f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.922112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c8457c-f604-439a-87e6-2f602da906f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.922396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c8457c-f604-439a-87e6-2f602da906f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.922668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c8457c-f604-439a-87e6-2f602da906f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.923327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c8457c-f604-439a-87e6-2f602da906f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.925285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c8457c-f604-439a-87e6-2f602da906f3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.926571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:33.927191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:33.927464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:33.927736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] PATH_INFO: `/v3/projects/296ce2199a01493dad9a7cc024f092f7` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:33.928930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] RBAC: Authorizing `identity:delete_project(project_id=296ce2199a01493dad9a7cc024f092f7)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:33.930805 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:33.936931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:33.954476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:33.957943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1329/2642] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:33 2022] POST /identity/v3/projects => generated 363 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:43:33.961934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c8457c-f604-439a-87e6-2f602da906f3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:33.981989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '296ce2199a01493dad9a7cc024f092f7', 'request_id': 'req-508c5866-ce5e-4303-b9b2-f41402848612'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.989045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '296ce2199a01493dad9a7cc024f092f7', 'request_id': 'req-508c5866-ce5e-4303-b9b2-f41402848612'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:33.997580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '296ce2199a01493dad9a7cc024f092f7', 'request_id': 'req-508c5866-ce5e-4303-b9b2-f41402848612'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:34.008215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] The token cache is being invalidate because project 296ce2199a01493dad9a7cc024f092f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:34.008551 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:34.009235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-508c5866-ce5e-4303-b9b2-f41402848612 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:34.011071 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1314/2643] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:33 2022] DELETE /identity/v3/projects/296ce2199a01493dad9a7cc024f092f7 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:34.016735 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e760e6f-928f-412e-98f6-ea4312f9ba16 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.017030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e760e6f-928f-412e-98f6-ea4312f9ba16 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.017299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e760e6f-928f-412e-98f6-ea4312f9ba16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.024522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c8457c-f604-439a-87e6-2f602da906f3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.037289 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7e760e6f-928f-412e-98f6-ea4312f9ba16 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:34.038509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7e760e6f-928f-412e-98f6-ea4312f9ba16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:34.090824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b13bbf-9759-401b-aa0f-e06bc104171d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.091430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b13bbf-9759-401b-aa0f-e06bc104171d admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.091715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b13bbf-9759-401b-aa0f-e06bc104171d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.091988 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b13bbf-9759-401b-aa0f-e06bc104171d admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.092653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b13bbf-9759-401b-aa0f-e06bc104171d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.095257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b13bbf-9759-401b-aa0f-e06bc104171d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.100544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1315/2644] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:34.109544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbffdb04-0b0d-4ef5-aa26-4eda91007b0c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.109838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbffdb04-0b0d-4ef5-aa26-4eda91007b0c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.110196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbffdb04-0b0d-4ef5-aa26-4eda91007b0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.118053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1330/2645] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:33 2022] POST /identity/v3/users => generated 288 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:43:34.122015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b13bbf-9759-401b-aa0f-e06bc104171d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.124056 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-47b13bbf-9759-401b-aa0f-e06bc104171d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:34.130030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fbffdb04-0b0d-4ef5-aa26-4eda91007b0c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:34.131235 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fbffdb04-0b0d-4ef5-aa26-4eda91007b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:34.136687 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b13bbf-9759-401b-aa0f-e06bc104171d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.139210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1316/2646] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:34.142430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25429a6-90f4-4c0b-86b0-48a04bc856b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.143372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25429a6-90f4-4c0b-86b0-48a04bc856b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.143760 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25429a6-90f4-4c0b-86b0-48a04bc856b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.143910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbffdb04-0b0d-4ef5-aa26-4eda91007b0c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.144165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25429a6-90f4-4c0b-86b0-48a04bc856b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.144940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25429a6-90f4-4c0b-86b0-48a04bc856b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.146596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25429a6-90f4-4c0b-86b0-48a04bc856b3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.152901 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbffdb04-0b0d-4ef5-aa26-4eda91007b0c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.160171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1331/2647] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:34 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:34.160991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a809d3f-be76-40ce-8228-74be872ad70e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.161145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25429a6-90f4-4c0b-86b0-48a04bc856b3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.161589 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a809d3f-be76-40ce-8228-74be872ad70e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.161870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a809d3f-be76-40ce-8228-74be872ad70e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.162141 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a809d3f-be76-40ce-8228-74be872ad70e admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.162782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a809d3f-be76-40ce-8228-74be872ad70e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:34.163088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a809d3f-be76-40ce-8228-74be872ad70e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.164716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a809d3f-be76-40ce-8228-74be872ad70e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.170524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25429a6-90f4-4c0b-86b0-48a04bc856b3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.176105 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056bffa9-7b27-4c18-8e96-a3ab70f2b664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.176561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1317/2648] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 26 01:43:34.176792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056bffa9-7b27-4c18-8e96-a3ab70f2b664 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.177078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056bffa9-7b27-4c18-8e96-a3ab70f2b664 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.177425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056bffa9-7b27-4c18-8e96-a3ab70f2b664 admin admin] PATH_INFO: `/v3/projects/0c6d5a1c6eb24527b0dd7cafc6a5feb6/users/b7f289f8983d4c269683a29c6d9c9a19/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.180952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a809d3f-be76-40ce-8228-74be872ad70e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.191493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a809d3f-be76-40ce-8228-74be872ad70e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.193148 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056bffa9-7b27-4c18-8e96-a3ab70f2b664 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c6d5a1c6eb24527b0dd7cafc6a5feb6, user_id=b7f289f8983d4c269683a29c6d9c9a19, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.195825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056bffa9-7b27-4c18-8e96-a3ab70f2b664 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.196296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731d8030-1e84-43e0-b816-64b3b0dcef5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.196900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731d8030-1e84-43e0-b816-64b3b0dcef5f admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.197180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731d8030-1e84-43e0-b816-64b3b0dcef5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.197448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731d8030-1e84-43e0-b816-64b3b0dcef5f admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.198060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731d8030-1e84-43e0-b816-64b3b0dcef5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.199963 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731d8030-1e84-43e0-b816-64b3b0dcef5f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.209421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1332/2649] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:34 2022] PUT /identity/v3/projects/0c6d5a1c6eb24527b0dd7cafc6a5feb6/users/b7f289f8983d4c269683a29c6d9c9a19/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:34.213889 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056bffa9-7b27-4c18-8e96-a3ab70f2b664 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.231188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1318/2650] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:34 2022] POST /identity/v3/projects => generated 375 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:43:34.232247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731d8030-1e84-43e0-b816-64b3b0dcef5f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.268091 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:34.279477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056bffa9-7b27-4c18-8e96-a3ab70f2b664 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.295481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731d8030-1e84-43e0-b816-64b3b0dcef5f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.300863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba896e7-1948-400c-9acf-f4b2f22440d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.301462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba896e7-1948-400c-9acf-f4b2f22440d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.301770 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba896e7-1948-400c-9acf-f4b2f22440d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.302045 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba896e7-1948-400c-9acf-f4b2f22440d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.302668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba896e7-1948-400c-9acf-f4b2f22440d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.304596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba896e7-1948-400c-9acf-f4b2f22440d5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.317126 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:34.328811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1319/2651] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:34 2022] POST /identity/v3/users => generated 288 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:34.332964 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba896e7-1948-400c-9acf-f4b2f22440d5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.342045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb609dde-bd4b-4632-b4ef-888c597f5fef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.342660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb609dde-bd4b-4632-b4ef-888c597f5fef admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.342938 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb609dde-bd4b-4632-b4ef-888c597f5fef admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.343069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba896e7-1948-400c-9acf-f4b2f22440d5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.343211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb609dde-bd4b-4632-b4ef-888c597f5fef admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.343997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb609dde-bd4b-4632-b4ef-888c597f5fef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.345944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb609dde-bd4b-4632-b4ef-888c597f5fef admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.348075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15e604b-1100-4006-a7c8-10c1ff1ea726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.348696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15e604b-1100-4006-a7c8-10c1ff1ea726 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.348976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15e604b-1100-4006-a7c8-10c1ff1ea726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.349254 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15e604b-1100-4006-a7c8-10c1ff1ea726 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.349895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15e604b-1100-4006-a7c8-10c1ff1ea726 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.351479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15e604b-1100-4006-a7c8-10c1ff1ea726 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.359999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1333/2652] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:34 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:34.361498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb609dde-bd4b-4632-b4ef-888c597f5fef admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.362621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1320/2653] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:34.366332 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15e604b-1100-4006-a7c8-10c1ff1ea726 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.371710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb609dde-bd4b-4632-b4ef-888c597f5fef admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.376255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15e604b-1100-4006-a7c8-10c1ff1ea726 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.376448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9031cda3-9511-4ad3-bde8-5384f85e6da3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.377027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9031cda3-9511-4ad3-bde8-5384f85e6da3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.377304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9031cda3-9511-4ad3-bde8-5384f85e6da3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.377575 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9031cda3-9511-4ad3-bde8-5384f85e6da3 admin admin] PATH_INFO: `/v3/projects/0c6d5a1c6eb24527b0dd7cafc6a5feb6/users/b7f289f8983d4c269683a29c6d9c9a19/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.379617 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9031cda3-9511-4ad3-bde8-5384f85e6da3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c6d5a1c6eb24527b0dd7cafc6a5feb6, user_id=b7f289f8983d4c269683a29c6d9c9a19, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.380995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e60501-689a-4e0d-b958-0f4de3dec54b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.381571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e60501-689a-4e0d-b958-0f4de3dec54b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.381851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e60501-689a-4e0d-b958-0f4de3dec54b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.382125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e60501-689a-4e0d-b958-0f4de3dec54b admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.382254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9031cda3-9511-4ad3-bde8-5384f85e6da3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.382764 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e60501-689a-4e0d-b958-0f4de3dec54b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.384330 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e60501-689a-4e0d-b958-0f4de3dec54b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.393932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1334/2654] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:34 2022] PUT /identity/v3/projects/0c6d5a1c6eb24527b0dd7cafc6a5feb6/users/b7f289f8983d4c269683a29c6d9c9a19/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:34.394792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1321/2655] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:34.398698 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9031cda3-9511-4ad3-bde8-5384f85e6da3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.405522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0f75d-78f0-4dd8-aa8c-058e6cd297f5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.405814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0f75d-78f0-4dd8-aa8c-058e6cd297f5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.406097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0f75d-78f0-4dd8-aa8c-058e6cd297f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.433545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9db0f75d-78f0-4dd8-aa8c-058e6cd297f5 None None] MFA Rules not processed for user `b7f289f8983d4c269683a29c6d9c9a19`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:34.434683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9db0f75d-78f0-4dd8-aa8c-058e6cd297f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:34.462273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9031cda3-9511-4ad3-bde8-5384f85e6da3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.467097 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3125e27-9ec7-420a-96ac-e979f0a4fbbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.467744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3125e27-9ec7-420a-96ac-e979f0a4fbbb admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.468023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3125e27-9ec7-420a-96ac-e979f0a4fbbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.468295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3125e27-9ec7-420a-96ac-e979f0a4fbbb admin admin] PATH_INFO: `/v3/projects/396de0ed39c34151b5043158317531e8/users/24e36230b3e54634961288bb8bafe708/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.482527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3125e27-9ec7-420a-96ac-e979f0a4fbbb admin admin] RBAC: Authorizing `identity:create_grant(project_id=396de0ed39c34151b5043158317531e8, user_id=24e36230b3e54634961288bb8bafe708, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.486018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3125e27-9ec7-420a-96ac-e979f0a4fbbb admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.498957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1335/2656] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:34 2022] PUT /identity/v3/projects/396de0ed39c34151b5043158317531e8/users/24e36230b3e54634961288bb8bafe708/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:34.509866 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce48dd30-b19e-4f19-9900-4e00638ec635 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.510152 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce48dd30-b19e-4f19-9900-4e00638ec635 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.510422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce48dd30-b19e-4f19-9900-4e00638ec635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.512038 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1322/2657] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:34 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:34.524192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db0f75d-78f0-4dd8-aa8c-058e6cd297f5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.525669 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9db0f75d-78f0-4dd8-aa8c-058e6cd297f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:34.542276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-ce48dd30-b19e-4f19-9900-4e00638ec635 None None] MFA Rules not processed for user `24e36230b3e54634961288bb8bafe708`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:34.543013 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ce48dd30-b19e-4f19-9900-4e00638ec635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:34.597091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db0f75d-78f0-4dd8-aa8c-058e6cd297f5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.632165 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:34.647822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1336/2658] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:34 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:34.655468 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce48dd30-b19e-4f19-9900-4e00638ec635 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:34.657458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ce48dd30-b19e-4f19-9900-4e00638ec635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:34.679488 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c78b5e5-232d-434a-bbfe-f0cbb49eae43 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.680255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c78b5e5-232d-434a-bbfe-f0cbb49eae43 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.680670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c78b5e5-232d-434a-bbfe-f0cbb49eae43 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.681042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c78b5e5-232d-434a-bbfe-f0cbb49eae43 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.730555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:34.743444 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce48dd30-b19e-4f19-9900-4e00638ec635 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:34.759828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6cb788-0abd-4cf6-92ae-bb8ec8de4029 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:34.760662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6cb788-0abd-4cf6-92ae-bb8ec8de4029 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:34.761059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6cb788-0abd-4cf6-92ae-bb8ec8de4029 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:34.761792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6cb788-0abd-4cf6-92ae-bb8ec8de4029 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:34.791566 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6cb788-0abd-4cf6-92ae-bb8ec8de4029 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.797316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6cb788-0abd-4cf6-92ae-bb8ec8de4029 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.806660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c78b5e5-232d-434a-bbfe-f0cbb49eae43 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:34.808394 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c78b5e5-232d-434a-bbfe-f0cbb49eae43 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:34.822988 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1323/2659] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:34 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:43:34.835142 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1337/2660] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:34 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:43:35.367220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c78b5e5-232d-434a-bbfe-f0cbb49eae43 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:35.439391 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c78b5e5-232d-434a-bbfe-f0cbb49eae43 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:35.443977 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:35.444636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:35.444924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:35.445230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] PATH_INFO: `/v3/users/8c517dedb30841a38e6139cdb9540e12` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:35.446351 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c517dedb30841a38e6139cdb9540e12)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:35.448265 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:35.499356 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c517dedb30841a38e6139cdb9540e12'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:35.508211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c517dedb30841a38e6139cdb9540e12'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:35.508700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c517dedb30841a38e6139cdb9540e12'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:35.509325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c517dedb30841a38e6139cdb9540e12'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:35.519160 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c517dedb30841a38e6139cdb9540e12'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:35.537191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1324/2661] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:35 2022] DELETE /identity/v3/users/8c517dedb30841a38e6139cdb9540e12 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:35.543293 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ad405-3b71-4e06-9d90-a5c71a6af21a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:35.543798 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ad405-3b71-4e06-9d90-a5c71a6af21a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:35.543916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ad405-3b71-4e06-9d90-a5c71a6af21a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:35.564372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-fa7ad405-3b71-4e06-9d90-a5c71a6af21a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:35.565715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fa7ad405-3b71-4e06-9d90-a5c71a6af21a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:35.628270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1338/2662] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:35.638140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:35.639568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:35.686037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:35.699700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46baf116-5641-49d2-8cee-59a4ed7ee543 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:35.703955 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a55ef9-1587-4d6a-b1f7-20c04938dfd3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:35.704579 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a55ef9-1587-4d6a-b1f7-20c04938dfd3 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:35.704858 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a55ef9-1587-4d6a-b1f7-20c04938dfd3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:35.705127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a55ef9-1587-4d6a-b1f7-20c04938dfd3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:35.714048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a55ef9-1587-4d6a-b1f7-20c04938dfd3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:35.715474 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a55ef9-1587-4d6a-b1f7-20c04938dfd3 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:35.724870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1325/2663] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:36.013323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7ad405-3b71-4e06-9d90-a5c71a6af21a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:36.014741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fa7ad405-3b71-4e06-9d90-a5c71a6af21a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:36.027193 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7ad405-3b71-4e06-9d90-a5c71a6af21a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:36.033432 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:36.034062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:36.034339 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:36.034929 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] PATH_INFO: `/v3/projects/25729b643dd943e482c777e4a6f2c9d6` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:36.035749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] RBAC: Authorizing `identity:delete_project(project_id=25729b643dd943e482c777e4a6f2c9d6)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:36.037696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:36.090805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '25729b643dd943e482c777e4a6f2c9d6', 'request_id': 'req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.097311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '25729b643dd943e482c777e4a6f2c9d6', 'request_id': 'req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.102813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '25729b643dd943e482c777e4a6f2c9d6', 'request_id': 'req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.115660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] The token cache is being invalidate because project 25729b643dd943e482c777e4a6f2c9d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:36.115947 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.116581 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.118051 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1339/2664] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:36 2022] DELETE /identity/v3/projects/25729b643dd943e482c777e4a6f2c9d6 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:36.122017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a55ef9-1587-4d6a-b1f7-20c04938dfd3 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:36.131890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a55ef9-1587-4d6a-b1f7-20c04938dfd3 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:36.187985 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:36.188655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:36.188932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:36.189479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] PATH_INFO: `/v3/users/b7f289f8983d4c269683a29c6d9c9a19` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:36.190613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7f289f8983d4c269683a29c6d9c9a19)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:36.192555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:36.237596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7f289f8983d4c269683a29c6d9c9a19'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.244671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7f289f8983d4c269683a29c6d9c9a19'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.245274 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7f289f8983d4c269683a29c6d9c9a19'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.245913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7f289f8983d4c269683a29c6d9c9a19'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.253469 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-014fd25c-a820-4c84-aa9c-e238232d7496 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7f289f8983d4c269683a29c6d9c9a19'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.271134 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1326/2665] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:36 2022] DELETE /identity/v3/users/b7f289f8983d4c269683a29c6d9c9a19 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:36.557837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:36.563251 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:36.584186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2e8df9-144d-41f3-a6af-0f71f6367ebe admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:36.606992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:36.670094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:36.670765 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:36.671047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:36.671319 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] PATH_INFO: `/v3/projects/0c6d5a1c6eb24527b0dd7cafc6a5feb6` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:36.672827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0c6d5a1c6eb24527b0dd7cafc6a5feb6)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:36.674749 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:36.729733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0c6d5a1c6eb24527b0dd7cafc6a5feb6', 'request_id': 'req-7adc46a0-ec67-484e-a784-fe790c1a59c2'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.737422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0c6d5a1c6eb24527b0dd7cafc6a5feb6', 'request_id': 'req-7adc46a0-ec67-484e-a784-fe790c1a59c2'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.743109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0c6d5a1c6eb24527b0dd7cafc6a5feb6', 'request_id': 'req-7adc46a0-ec67-484e-a784-fe790c1a59c2'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.764441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] The token cache is being invalidate because project 0c6d5a1c6eb24527b0dd7cafc6a5feb6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:36.764741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.765952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7adc46a0-ec67-484e-a784-fe790c1a59c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:36.767678 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1340/2666] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:36 2022] DELETE /identity/v3/projects/0c6d5a1c6eb24527b0dd7cafc6a5feb6 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:36.772449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fe2228-070e-433a-a374-40966ed5cb47 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:36.772958 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fe2228-070e-433a-a374-40966ed5cb47 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:36.773050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fe2228-070e-433a-a374-40966ed5cb47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:36.794259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-36fe2228-070e-433a-a374-40966ed5cb47 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:36.795475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-36fe2228-070e-433a-a374-40966ed5cb47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:36.861097 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1327/2667] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:36.869073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300bdb34-970a-4a97-af6c-10b0a04f7293 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:36.869356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300bdb34-970a-4a97-af6c-10b0a04f7293 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:36.869628 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300bdb34-970a-4a97-af6c-10b0a04f7293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:36.895921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-300bdb34-970a-4a97-af6c-10b0a04f7293 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:36.895921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-300bdb34-970a-4a97-af6c-10b0a04f7293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:36.900728 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1341/2668] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:36.904831 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fe2228-070e-433a-a374-40966ed5cb47 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:36.914794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fe2228-070e-433a-a374-40966ed5cb47 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:36.919375 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d23391-45d4-421e-a8a0-c71c79ed105d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:36.919970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d23391-45d4-421e-a8a0-c71c79ed105d admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:36.920248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d23391-45d4-421e-a8a0-c71c79ed105d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:36.920555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d23391-45d4-421e-a8a0-c71c79ed105d admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:36.921198 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d23391-45d4-421e-a8a0-c71c79ed105d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:36.921500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d23391-45d4-421e-a8a0-c71c79ed105d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:36.923149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d23391-45d4-421e-a8a0-c71c79ed105d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:36.924671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:36.939504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1328/2669] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:43:36.943557 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300bdb34-970a-4a97-af6c-10b0a04f7293 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:36.954285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300bdb34-970a-4a97-af6c-10b0a04f7293 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:36.959308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bac0db2-225d-4297-aa2f-35a85b277002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:36.960010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bac0db2-225d-4297-aa2f-35a85b277002 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:36.960371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bac0db2-225d-4297-aa2f-35a85b277002 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:36.960731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bac0db2-225d-4297-aa2f-35a85b277002 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:36.961553 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bac0db2-225d-4297-aa2f-35a85b277002 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:36.963624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bac0db2-225d-4297-aa2f-35a85b277002 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:36.998998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1342/2670] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:36 2022] POST /identity/v3/projects => generated 383 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:43:37.008544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d23391-45d4-421e-a8a0-c71c79ed105d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:37.018946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d23391-45d4-421e-a8a0-c71c79ed105d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:37.077896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf74452-490b-4bd2-abb8-9d526a01a376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:37.079249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf74452-490b-4bd2-abb8-9d526a01a376 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:37.079540 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf74452-490b-4bd2-abb8-9d526a01a376 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:37.079817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf74452-490b-4bd2-abb8-9d526a01a376 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:37.080494 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf74452-490b-4bd2-abb8-9d526a01a376 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:37.082415 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf74452-490b-4bd2-abb8-9d526a01a376 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:37.106949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1329/2671] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:37 2022] POST /identity/v3/users => generated 292 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:37.111800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bac0db2-225d-4297-aa2f-35a85b277002 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:37.124585 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bac0db2-225d-4297-aa2f-35a85b277002 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:37.130129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57600eb7-8721-4cc8-99bb-438b317006c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:37.130957 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57600eb7-8721-4cc8-99bb-438b317006c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:37.131433 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57600eb7-8721-4cc8-99bb-438b317006c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:37.131719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57600eb7-8721-4cc8-99bb-438b317006c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:37.132419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57600eb7-8721-4cc8-99bb-438b317006c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:37.134183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57600eb7-8721-4cc8-99bb-438b317006c9 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:37.145807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1343/2672] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:37 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:37.151159 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf74452-490b-4bd2-abb8-9d526a01a376 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:37.162370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf74452-490b-4bd2-abb8-9d526a01a376 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:37.169531 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f7631-e0c2-414b-87bf-4034a8fd5a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:37.170275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f7631-e0c2-414b-87bf-4034a8fd5a90 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:37.170561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f7631-e0c2-414b-87bf-4034a8fd5a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:37.170838 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f7631-e0c2-414b-87bf-4034a8fd5a90 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:37.172255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f7631-e0c2-414b-87bf-4034a8fd5a90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:37.174195 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f7631-e0c2-414b-87bf-4034a8fd5a90 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:37.186001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1330/2673] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:37 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:37.191258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57600eb7-8721-4cc8-99bb-438b317006c9 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:37.202360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57600eb7-8721-4cc8-99bb-438b317006c9 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:37.208619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47129300-71f7-4576-a80b-134045491827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:37.209278 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47129300-71f7-4576-a80b-134045491827 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:37.209554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47129300-71f7-4576-a80b-134045491827 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:37.209824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47129300-71f7-4576-a80b-134045491827 admin admin] PATH_INFO: `/v3/projects/c66d53ff01814d74900b6086fc71c874/users/74b21bcb052e464ba171a0bea6c8b9a6/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:37.226084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47129300-71f7-4576-a80b-134045491827 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c66d53ff01814d74900b6086fc71c874, user_id=74b21bcb052e464ba171a0bea6c8b9a6, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:37.228961 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47129300-71f7-4576-a80b-134045491827 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:37.244090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1344/2674] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:37 2022] PUT /identity/v3/projects/c66d53ff01814d74900b6086fc71c874/users/74b21bcb052e464ba171a0bea6c8b9a6/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:37.254571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7bd21c-9f07-4901-bcbd-5ee160160d76 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:37.254871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7bd21c-9f07-4901-bcbd-5ee160160d76 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:37.255136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7bd21c-9f07-4901-bcbd-5ee160160d76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:37.285082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ad7bd21c-9f07-4901-bcbd-5ee160160d76 None None] MFA Rules not processed for user `74b21bcb052e464ba171a0bea6c8b9a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:37.286376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ad7bd21c-9f07-4901-bcbd-5ee160160d76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:37.354296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1331/2675] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:37 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:37.365888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47129300-71f7-4576-a80b-134045491827 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:37.367572 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-47129300-71f7-4576-a80b-134045491827 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:37.422832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:37.427119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47129300-71f7-4576-a80b-134045491827 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:37.433197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76822c4-0af6-4608-81ea-ff09f01ba8bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:37.433976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76822c4-0af6-4608-81ea-ff09f01ba8bc service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:37.434496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76822c4-0af6-4608-81ea-ff09f01ba8bc service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:37.434825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76822c4-0af6-4608-81ea-ff09f01ba8bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:37.445004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76822c4-0af6-4608-81ea-ff09f01ba8bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:37.447130 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76822c4-0af6-4608-81ea-ff09f01ba8bc service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:37.456374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1345/2676] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:37 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:37.475941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7bd21c-9f07-4901-bcbd-5ee160160d76 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:37.538794 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7bd21c-9f07-4901-bcbd-5ee160160d76 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:37.543691 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:37.544313 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:37.544621 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:37.544894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] PATH_INFO: `/v3/users/74b21bcb052e464ba171a0bea6c8b9a6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:37.546043 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] RBAC: Authorizing `identity:delete_user(user_id=74b21bcb052e464ba171a0bea6c8b9a6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:37.548497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:37.550904 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:37.596682 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74b21bcb052e464ba171a0bea6c8b9a6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:37.604433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74b21bcb052e464ba171a0bea6c8b9a6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:37.605064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74b21bcb052e464ba171a0bea6c8b9a6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:37.605716 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74b21bcb052e464ba171a0bea6c8b9a6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:37.613535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74b21bcb052e464ba171a0bea6c8b9a6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:37.632453 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1332/2677] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:37 2022] DELETE /identity/v3/users/74b21bcb052e464ba171a0bea6c8b9a6 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:37.636985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb1c2d-aa94-41e9-93cf-f621b82fc145 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:37.637454 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb1c2d-aa94-41e9-93cf-f621b82fc145 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:37.637631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edb1c2d-aa94-41e9-93cf-f621b82fc145 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:37.659462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0edb1c2d-aa94-41e9-93cf-f621b82fc145 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:37.660720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0edb1c2d-aa94-41e9-93cf-f621b82fc145 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:37.721721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1346/2678] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:37.735506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:37.737057 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:37.795671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab06a87-4ea6-4bd8-8394-411eaec6fc49 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:37.800512 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5a4fa3-f972-4dae-88fb-8a57bc223edc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:37.801108 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a4fa3-f972-4dae-88fb-8a57bc223edc service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:37.801387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a4fa3-f972-4dae-88fb-8a57bc223edc service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:37.801647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a4fa3-f972-4dae-88fb-8a57bc223edc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:37.811199 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5a4fa3-f972-4dae-88fb-8a57bc223edc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:37.812966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5a4fa3-f972-4dae-88fb-8a57bc223edc service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:37.822830 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1333/2679] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:38.046872 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edb1c2d-aa94-41e9-93cf-f621b82fc145 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:38.048590 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0edb1c2d-aa94-41e9-93cf-f621b82fc145 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:38.062170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edb1c2d-aa94-41e9-93cf-f621b82fc145 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:38.067691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:38.068337 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.068644 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.068923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] PATH_INFO: `/v3/users/24e36230b3e54634961288bb8bafe708` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.070112 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] RBAC: Authorizing `identity:delete_user(user_id=24e36230b3e54634961288bb8bafe708)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:38.072086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:38.122035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24e36230b3e54634961288bb8bafe708'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.131197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24e36230b3e54634961288bb8bafe708'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.131865 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24e36230b3e54634961288bb8bafe708'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.132501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24e36230b3e54634961288bb8bafe708'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.146529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24e36230b3e54634961288bb8bafe708'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.161183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1347/2680] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:38 2022] DELETE /identity/v3/users/24e36230b3e54634961288bb8bafe708 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:38.165728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171233e1-7c83-4d8f-83e6-adc2feb5117d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.166656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171233e1-7c83-4d8f-83e6-adc2feb5117d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.166656 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171233e1-7c83-4d8f-83e6-adc2feb5117d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.186683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-171233e1-7c83-4d8f-83e6-adc2feb5117d None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:38.188524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-171233e1-7c83-4d8f-83e6-adc2feb5117d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:38.259538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1334/2681] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:38.279159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:38.280841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:38.318307 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171233e1-7c83-4d8f-83e6-adc2feb5117d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:38.319844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-171233e1-7c83-4d8f-83e6-adc2feb5117d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:38.347387 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171233e1-7c83-4d8f-83e6-adc2feb5117d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:38.353519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6532532-28c4-467d-b5f8-d7d8c28e3d04 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:38.355677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:38.357121 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.357506 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.357810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c024e81-93bf-4b79-9e53-713cd3d61f12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:38.357939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] PATH_INFO: `/v3/projects/c66d53ff01814d74900b6086fc71c874` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.358410 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c024e81-93bf-4b79-9e53-713cd3d61f12 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.358735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c024e81-93bf-4b79-9e53-713cd3d61f12 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.359037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c024e81-93bf-4b79-9e53-713cd3d61f12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.360617 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c66d53ff01814d74900b6086fc71c874)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:38.363587 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:38.370856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c024e81-93bf-4b79-9e53-713cd3d61f12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:38.373389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c024e81-93bf-4b79-9e53-713cd3d61f12 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:38.390124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1348/2682] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:38.393814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:38.410740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c024e81-93bf-4b79-9e53-713cd3d61f12 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:38.414030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0c024e81-93bf-4b79-9e53-713cd3d61f12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:38.429734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c66d53ff01814d74900b6086fc71c874', 'request_id': 'req-071d1681-cccc-4c86-bd79-5539255b7ce7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.441980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:38.442226 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c66d53ff01814d74900b6086fc71c874', 'request_id': 'req-071d1681-cccc-4c86-bd79-5539255b7ce7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.459700 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c66d53ff01814d74900b6086fc71c874', 'request_id': 'req-071d1681-cccc-4c86-bd79-5539255b7ce7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.480930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] The token cache is being invalidate because project c66d53ff01814d74900b6086fc71c874 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:38.481775 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.483174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071d1681-cccc-4c86-bd79-5539255b7ce7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.486245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1335/2683] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:38 2022] DELETE /identity/v3/projects/c66d53ff01814d74900b6086fc71c874 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:38.491736 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda1bb0e-9479-4fba-85a8-5f96255c8bf3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.492128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda1bb0e-9479-4fba-85a8-5f96255c8bf3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.492593 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda1bb0e-9479-4fba-85a8-5f96255c8bf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.524536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-eda1bb0e-9479-4fba-85a8-5f96255c8bf3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:38.526353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-eda1bb0e-9479-4fba-85a8-5f96255c8bf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:38.561767 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c024e81-93bf-4b79-9e53-713cd3d61f12 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:38.613937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1336/2684] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:38.621730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa13016e-9d71-4cd9-ae4c-4ed1659feacc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.622015 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa13016e-9d71-4cd9-ae4c-4ed1659feacc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.622291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa13016e-9d71-4cd9-ae4c-4ed1659feacc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.647042 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-aa13016e-9d71-4cd9-ae4c-4ed1659feacc None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:38.648635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-aa13016e-9d71-4cd9-ae4c-4ed1659feacc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:38.658959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1337/2685] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:38.663555 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa13016e-9d71-4cd9-ae4c-4ed1659feacc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:38.674588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa13016e-9d71-4cd9-ae4c-4ed1659feacc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:38.676831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410ea6b7-5f87-444c-8e8a-6c40aa919948 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:38.677456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410ea6b7-5f87-444c-8e8a-6c40aa919948 service nova] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.677949 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410ea6b7-5f87-444c-8e8a-6c40aa919948 service nova] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.678239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410ea6b7-5f87-444c-8e8a-6c40aa919948 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.680685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-410ea6b7-5f87-444c-8e8a-6c40aa919948 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:38.680981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea589105-9e54-4793-bb02-f7073f4bc3fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:38.681772 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea589105-9e54-4793-bb02-f7073f4bc3fc admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.682204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea589105-9e54-4793-bb02-f7073f4bc3fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.682628 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea589105-9e54-4793-bb02-f7073f4bc3fc admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.683433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea589105-9e54-4793-bb02-f7073f4bc3fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:38.684040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea589105-9e54-4793-bb02-f7073f4bc3fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:38.685844 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea589105-9e54-4793-bb02-f7073f4bc3fc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:38.700001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410ea6b7-5f87-444c-8e8a-6c40aa919948 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:38.701900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1338/2686] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:43:38.707741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410ea6b7-5f87-444c-8e8a-6c40aa919948 service nova] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:38.709123 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea589105-9e54-4793-bb02-f7073f4bc3fc admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:38.722498 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea589105-9e54-4793-bb02-f7073f4bc3fc admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:38.727609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1349/2687] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:38.729130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4478e538-b271-4f07-8e98-e38d83d25932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:38.729782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4478e538-b271-4f07-8e98-e38d83d25932 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.730064 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4478e538-b271-4f07-8e98-e38d83d25932 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.730334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4478e538-b271-4f07-8e98-e38d83d25932 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.731044 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4478e538-b271-4f07-8e98-e38d83d25932 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:38.733448 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4478e538-b271-4f07-8e98-e38d83d25932 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:38.743938 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410ea6b7-5f87-444c-8e8a-6c40aa919948 service nova] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:38.745910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-410ea6b7-5f87-444c-8e8a-6c40aa919948 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:38.768138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1339/2688] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:38 2022] POST /identity/v3/projects => generated 385 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:38.772295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4478e538-b271-4f07-8e98-e38d83d25932 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:38.782303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410ea6b7-5f87-444c-8e8a-6c40aa919948 service nova] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:38.860890 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:38.861562 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.861827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.862104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] PATH_INFO: `/v3/projects/396de0ed39c34151b5043158317531e8` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.863512 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=396de0ed39c34151b5043158317531e8)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:38.865499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:38.883615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4478e538-b271-4f07-8e98-e38d83d25932 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:38.888743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9f48e7-7965-4b97-b234-e53d68fd96c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:38.889347 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9f48e7-7965-4b97-b234-e53d68fd96c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.889624 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9f48e7-7965-4b97-b234-e53d68fd96c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.889894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9f48e7-7965-4b97-b234-e53d68fd96c9 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:38.891183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9f48e7-7965-4b97-b234-e53d68fd96c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:38.893686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9f48e7-7965-4b97-b234-e53d68fd96c9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:38.907966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:38.913015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:38.927557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1340/2689] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:38 2022] POST /identity/v3/users => generated 293 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:38.930576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9f48e7-7965-4b97-b234-e53d68fd96c9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:38.937911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '396de0ed39c34151b5043158317531e8', 'request_id': 'req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.943101 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9f48e7-7965-4b97-b234-e53d68fd96c9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:38.947079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '396de0ed39c34151b5043158317531e8', 'request_id': 'req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.954829 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '396de0ed39c34151b5043158317531e8', 'request_id': 'req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.976981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] The token cache is being invalidate because project 396de0ed39c34151b5043158317531e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:38.977386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.978507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9956b105-41ed-49b2-a6d7-01f1f4fc6a56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:38.980121 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1350/2690] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:38 2022] DELETE /identity/v3/projects/396de0ed39c34151b5043158317531e8 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:38.985621 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c2b45f-889b-486d-b3a9-8e3a0202090a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:38.987045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c2b45f-889b-486d-b3a9-8e3a0202090a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:38.987407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c2b45f-889b-486d-b3a9-8e3a0202090a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.010871 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e7c2b45f-889b-486d-b3a9-8e3a0202090a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:39.012719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e7c2b45f-889b-486d-b3a9-8e3a0202090a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:39.025337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6f4268-8ad9-45e5-82fb-07b361e144cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.026002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6f4268-8ad9-45e5-82fb-07b361e144cb admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.026535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6f4268-8ad9-45e5-82fb-07b361e144cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.026893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6f4268-8ad9-45e5-82fb-07b361e144cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.027679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6f4268-8ad9-45e5-82fb-07b361e144cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.030007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6f4268-8ad9-45e5-82fb-07b361e144cb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.044215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1341/2691] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:38 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:39.049026 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6f4268-8ad9-45e5-82fb-07b361e144cb admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:39.054631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2a6f4268-8ad9-45e5-82fb-07b361e144cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:39.070399 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6f4268-8ad9-45e5-82fb-07b361e144cb admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:39.079435 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85171da-ead1-48ae-9f6b-d1847c925f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.081503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85171da-ead1-48ae-9f6b-d1847c925f7b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.082329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85171da-ead1-48ae-9f6b-d1847c925f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.083322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85171da-ead1-48ae-9f6b-d1847c925f7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.084712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85171da-ead1-48ae-9f6b-d1847c925f7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.086777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85171da-ead1-48ae-9f6b-d1847c925f7b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.101982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1342/2692] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:39 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:39.106216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1351/2693] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:39.107324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85171da-ead1-48ae-9f6b-d1847c925f7b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:39.117117 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58097923-e828-404d-b52f-fc253c3848d6 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.117769 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58097923-e828-404d-b52f-fc253c3848d6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.118277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58097923-e828-404d-b52f-fc253c3848d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.122263 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85171da-ead1-48ae-9f6b-d1847c925f7b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:39.129260 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f47e50-96e2-4a9a-9d19-dfb0e62c86d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.129951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f47e50-96e2-4a9a-9d19-dfb0e62c86d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.130230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f47e50-96e2-4a9a-9d19-dfb0e62c86d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.130505 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f47e50-96e2-4a9a-9d19-dfb0e62c86d1 admin admin] PATH_INFO: `/v3/projects/6a484d5ff7e249cb96c342c2358ada67/users/2dead56318924aa7aa4d3acf883c6369/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.142813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-58097923-e828-404d-b52f-fc253c3848d6 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:39.144232 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-58097923-e828-404d-b52f-fc253c3848d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:39.149972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f47e50-96e2-4a9a-9d19-dfb0e62c86d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6a484d5ff7e249cb96c342c2358ada67, user_id=2dead56318924aa7aa4d3acf883c6369, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.153141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1352/2694] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:39.155218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f47e50-96e2-4a9a-9d19-dfb0e62c86d1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.157500 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58097923-e828-404d-b52f-fc253c3848d6 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:39.168073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58097923-e828-404d-b52f-fc253c3848d6 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:39.175252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1343/2695] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:39 2022] PUT /identity/v3/projects/6a484d5ff7e249cb96c342c2358ada67/users/2dead56318924aa7aa4d3acf883c6369/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:39.185984 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b343482a-dfed-4b83-8458-9d142bc49d1c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.186702 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b343482a-dfed-4b83-8458-9d142bc49d1c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.187034 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b343482a-dfed-4b83-8458-9d142bc49d1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.223517 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b343482a-dfed-4b83-8458-9d142bc49d1c None None] MFA Rules not processed for user `2dead56318924aa7aa4d3acf883c6369`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:39.224823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b343482a-dfed-4b83-8458-9d142bc49d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:39.244561 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b15e5d-3dc4-4a0e-8822-5d34dc47c9ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.245336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b15e5d-3dc4-4a0e-8822-5d34dc47c9ae admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.246009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b15e5d-3dc4-4a0e-8822-5d34dc47c9ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.246172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b15e5d-3dc4-4a0e-8822-5d34dc47c9ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.246841 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b15e5d-3dc4-4a0e-8822-5d34dc47c9ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:39.247150 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b15e5d-3dc4-4a0e-8822-5d34dc47c9ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.249103 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b15e5d-3dc4-4a0e-8822-5d34dc47c9ae admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.276088 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1353/2696] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:43:39.280650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b15e5d-3dc4-4a0e-8822-5d34dc47c9ae admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:39.294462 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b15e5d-3dc4-4a0e-8822-5d34dc47c9ae admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:39.302707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aa3574-26bb-4ae6-be1a-9a6ced2da411 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.303887 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aa3574-26bb-4ae6-be1a-9a6ced2da411 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.304141 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1344/2697] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:39.304668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aa3574-26bb-4ae6-be1a-9a6ced2da411 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.305182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aa3574-26bb-4ae6-be1a-9a6ced2da411 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.306524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10aa3574-26bb-4ae6-be1a-9a6ced2da411 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.308691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10aa3574-26bb-4ae6-be1a-9a6ced2da411 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.311420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b343482a-dfed-4b83-8458-9d142bc49d1c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:39.314435 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b343482a-dfed-4b83-8458-9d142bc49d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:39.348122 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1354/2698] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:39 2022] POST /identity/v3/projects => generated 375 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:39.352419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aa3574-26bb-4ae6-be1a-9a6ced2da411 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:39.389712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:39.398863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b343482a-dfed-4b83-8458-9d142bc49d1c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:39.409202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:39.461300 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aa3574-26bb-4ae6-be1a-9a6ced2da411 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:39.468180 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f97779-67a0-4640-b4d3-119fb64f0920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.469287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f97779-67a0-4640-b4d3-119fb64f0920 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.469663 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f97779-67a0-4640-b4d3-119fb64f0920 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.469793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9ab961-bf7a-4203-a8e0-a7e8806479c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.470257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f97779-67a0-4640-b4d3-119fb64f0920 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.470445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9ab961-bf7a-4203-a8e0-a7e8806479c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.470734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9ab961-bf7a-4203-a8e0-a7e8806479c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.471020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9ab961-bf7a-4203-a8e0-a7e8806479c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.471243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f97779-67a0-4640-b4d3-119fb64f0920 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.473523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f97779-67a0-4640-b4d3-119fb64f0920 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.499960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1355/2699] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:39 2022] POST /identity/v3/users => generated 288 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:43:39.505446 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f97779-67a0-4640-b4d3-119fb64f0920 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:39.519921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f97779-67a0-4640-b4d3-119fb64f0920 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:39.525343 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6039930a-8f03-46ab-b14a-aa6942e07d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.525943 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039930a-8f03-46ab-b14a-aa6942e07d7f admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.526248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039930a-8f03-46ab-b14a-aa6942e07d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.526623 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6039930a-8f03-46ab-b14a-aa6942e07d7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.527271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6039930a-8f03-46ab-b14a-aa6942e07d7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.528904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6039930a-8f03-46ab-b14a-aa6942e07d7f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.541197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1356/2700] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:39 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:39.542910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9ab961-bf7a-4203-a8e0-a7e8806479c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.544282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6039930a-8f03-46ab-b14a-aa6942e07d7f admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:39.544626 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9ab961-bf7a-4203-a8e0-a7e8806479c3 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.555205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6039930a-8f03-46ab-b14a-aa6942e07d7f admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:39.555497 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1345/2701] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:43:39.560704 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca980d6-0d33-47f7-b4b5-fe77aece9e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.561485 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca980d6-0d33-47f7-b4b5-fe77aece9e1d admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.561884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca980d6-0d33-47f7-b4b5-fe77aece9e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.562270 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca980d6-0d33-47f7-b4b5-fe77aece9e1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.563174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca980d6-0d33-47f7-b4b5-fe77aece9e1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.565336 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca980d6-0d33-47f7-b4b5-fe77aece9e1d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.578896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1357/2702] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:39 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:39.583222 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9ab961-bf7a-4203-a8e0-a7e8806479c3 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:39.594719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9ab961-bf7a-4203-a8e0-a7e8806479c3 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:39.599504 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c572d2d-9fb8-49bf-90ec-f0a51d29e948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.600386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c572d2d-9fb8-49bf-90ec-f0a51d29e948 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.600513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c572d2d-9fb8-49bf-90ec-f0a51d29e948 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.600746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c572d2d-9fb8-49bf-90ec-f0a51d29e948 admin admin] PATH_INFO: `/v3/projects/10895902ec774f7ab15db01ce507471d/users/21d71eca7714406d8f97c061d6b26ce9/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.618219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c572d2d-9fb8-49bf-90ec-f0a51d29e948 admin admin] RBAC: Authorizing `identity:create_grant(project_id=10895902ec774f7ab15db01ce507471d, user_id=21d71eca7714406d8f97c061d6b26ce9, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.622712 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c572d2d-9fb8-49bf-90ec-f0a51d29e948 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.638880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1346/2703] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:39 2022] PUT /identity/v3/projects/10895902ec774f7ab15db01ce507471d/users/21d71eca7714406d8f97c061d6b26ce9/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:39.651147 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0817d22-cb55-48fc-af0c-8e0722f49c9b None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.651354 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0817d22-cb55-48fc-af0c-8e0722f49c9b None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.651625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0817d22-cb55-48fc-af0c-8e0722f49c9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.687714 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-f0817d22-cb55-48fc-af0c-8e0722f49c9b None None] MFA Rules not processed for user `21d71eca7714406d8f97c061d6b26ce9`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:39.689498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-f0817d22-cb55-48fc-af0c-8e0722f49c9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:39.766901 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1358/2704] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:39 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:39.777637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c572d2d-9fb8-49bf-90ec-f0a51d29e948 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:39.779422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3c572d2d-9fb8-49bf-90ec-f0a51d29e948 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:39.848105 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c572d2d-9fb8-49bf-90ec-f0a51d29e948 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:39.853862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6aced-e1d8-4d14-a8e2-06e4cb06144a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:39.854726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6aced-e1d8-4d14-a8e2-06e4cb06144a service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:39.855236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6aced-e1d8-4d14-a8e2-06e4cb06144a service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:39.856817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6aced-e1d8-4d14-a8e2-06e4cb06144a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:39.869431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf6aced-e1d8-4d14-a8e2-06e4cb06144a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:39.871107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf6aced-e1d8-4d14-a8e2-06e4cb06144a service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:39.888302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1347/2705] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:39 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:41.270452 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8488efb-bd1b-46ad-9e85-ad6b7f751a37 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:41.271037 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8488efb-bd1b-46ad-9e85-ad6b7f751a37 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:41.271186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8488efb-bd1b-46ad-9e85-ad6b7f751a37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:41.290881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-c8488efb-bd1b-46ad-9e85-ad6b7f751a37 None None] MFA Rules not processed for user `21d71eca7714406d8f97c061d6b26ce9`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:41.292094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c8488efb-bd1b-46ad-9e85-ad6b7f751a37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:41.299104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1359/2706] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:41 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:42.178780 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6aced-e1d8-4d14-a8e2-06e4cb06144a service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:42.220500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:42.245293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6aced-e1d8-4d14-a8e2-06e4cb06144a service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:42.250526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:42.251147 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:42.251416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:42.251686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] PATH_INFO: `/v3/users/2dead56318924aa7aa4d3acf883c6369` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:42.252834 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2dead56318924aa7aa4d3acf883c6369)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:42.254762 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:42.304246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2dead56318924aa7aa4d3acf883c6369'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:42.311865 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2dead56318924aa7aa4d3acf883c6369'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:42.313216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2dead56318924aa7aa4d3acf883c6369'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:42.314083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2dead56318924aa7aa4d3acf883c6369'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:42.322288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2dead56318924aa7aa4d3acf883c6369'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:42.340047 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1348/2707] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:42 2022] DELETE /identity/v3/users/2dead56318924aa7aa4d3acf883c6369 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:42.345431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a773dae-6666-473d-b16f-c9fe2e1a0abf None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:42.346009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a773dae-6666-473d-b16f-c9fe2e1a0abf None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:42.346009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a773dae-6666-473d-b16f-c9fe2e1a0abf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:42.365799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-5a773dae-6666-473d-b16f-c9fe2e1a0abf None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:42.366999 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5a773dae-6666-473d-b16f-c9fe2e1a0abf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:42.429448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1360/2708] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:42.439290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:42.440728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:42.499653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55c4521-4765-4977-8605-86964494ff8d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:42.503272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ac75b2-70ba-4c7e-b3f8-2703b6f2475a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:42.503888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ac75b2-70ba-4c7e-b3f8-2703b6f2475a service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:42.504162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ac75b2-70ba-4c7e-b3f8-2703b6f2475a service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:42.504458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ac75b2-70ba-4c7e-b3f8-2703b6f2475a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:42.513237 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ac75b2-70ba-4c7e-b3f8-2703b6f2475a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:42.514649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ac75b2-70ba-4c7e-b3f8-2703b6f2475a service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:42.524819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1349/2709] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:42.793341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a773dae-6666-473d-b16f-c9fe2e1a0abf None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:42.794874 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5a773dae-6666-473d-b16f-c9fe2e1a0abf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:42.831795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a773dae-6666-473d-b16f-c9fe2e1a0abf None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:42.839602 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:42.840587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:42.840978 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:42.841846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] PATH_INFO: `/v3/projects/6a484d5ff7e249cb96c342c2358ada67` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:42.843160 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6a484d5ff7e249cb96c342c2358ada67)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:42.846814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:42.862275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:42.915299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6a484d5ff7e249cb96c342c2358ada67', 'request_id': 'req-964c3619-cc09-4f0c-b548-57a8ae865102'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:42.923196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6a484d5ff7e249cb96c342c2358ada67', 'request_id': 'req-964c3619-cc09-4f0c-b548-57a8ae865102'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:42.930115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6a484d5ff7e249cb96c342c2358ada67', 'request_id': 'req-964c3619-cc09-4f0c-b548-57a8ae865102'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:42.959937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] The token cache is being invalidate because project 6a484d5ff7e249cb96c342c2358ada67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:42.959937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:42.959937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-964c3619-cc09-4f0c-b548-57a8ae865102 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:42.959937 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1361/2710] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:42 2022] DELETE /identity/v3/projects/6a484d5ff7e249cb96c342c2358ada67 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:42.965197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8c81c1-e6ae-4e7e-a65f-dc3c18dfc4b6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:42.965197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8c81c1-e6ae-4e7e-a65f-dc3c18dfc4b6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:42.965197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8c81c1-e6ae-4e7e-a65f-dc3c18dfc4b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:42.985066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5b8c81c1-e6ae-4e7e-a65f-dc3c18dfc4b6 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:42.986300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5b8c81c1-e6ae-4e7e-a65f-dc3c18dfc4b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:43.047742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:43.059459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1350/2711] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:43.067731 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe646ad-bae5-4879-8822-55844e4bdb14 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.068041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe646ad-bae5-4879-8822-55844e4bdb14 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.068316 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe646ad-bae5-4879-8822-55844e4bdb14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.089879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-5fe646ad-bae5-4879-8822-55844e4bdb14 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:43.091094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5fe646ad-bae5-4879-8822-55844e4bdb14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:43.099863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1362/2712] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:43.104315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8c81c1-e6ae-4e7e-a65f-dc3c18dfc4b6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:43.114634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8c81c1-e6ae-4e7e-a65f-dc3c18dfc4b6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:43.119708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc06718-2845-4843-90c5-36168566d60d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:43.120329 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc06718-2845-4843-90c5-36168566d60d admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.120641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc06718-2845-4843-90c5-36168566d60d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.120915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc06718-2845-4843-90c5-36168566d60d admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.121568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc06718-2845-4843-90c5-36168566d60d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:43.121875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc06718-2845-4843-90c5-36168566d60d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:43.123596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc06718-2845-4843-90c5-36168566d60d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:43.136001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1351/2713] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:43:43.140745 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe646ad-bae5-4879-8822-55844e4bdb14 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:43.150506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe646ad-bae5-4879-8822-55844e4bdb14 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:43.155180 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de313297-10fa-4301-93d9-adf3a35c31fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:43.156325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de313297-10fa-4301-93d9-adf3a35c31fd admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.156828 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de313297-10fa-4301-93d9-adf3a35c31fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.157277 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de313297-10fa-4301-93d9-adf3a35c31fd admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.157920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de313297-10fa-4301-93d9-adf3a35c31fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:43.159829 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de313297-10fa-4301-93d9-adf3a35c31fd admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:43.192473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1363/2714] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:43 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:43.196832 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc06718-2845-4843-90c5-36168566d60d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:43.207837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc06718-2845-4843-90c5-36168566d60d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:43.265980 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f064353a-69ab-4d13-a308-5fc4214045dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:43.266585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f064353a-69ab-4d13-a308-5fc4214045dc admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.267061 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f064353a-69ab-4d13-a308-5fc4214045dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.267339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f064353a-69ab-4d13-a308-5fc4214045dc admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.267968 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f064353a-69ab-4d13-a308-5fc4214045dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:43.269991 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f064353a-69ab-4d13-a308-5fc4214045dc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:43.292566 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1352/2715] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:43 2022] POST /identity/v3/users => generated 287 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:43.296551 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de313297-10fa-4301-93d9-adf3a35c31fd admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:43.307118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de313297-10fa-4301-93d9-adf3a35c31fd admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:43.312019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28afc34c-2c4c-4074-8761-e162f5d2f138 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:43.312715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28afc34c-2c4c-4074-8761-e162f5d2f138 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.312988 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28afc34c-2c4c-4074-8761-e162f5d2f138 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.313258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28afc34c-2c4c-4074-8761-e162f5d2f138 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.314167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28afc34c-2c4c-4074-8761-e162f5d2f138 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:43.315799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28afc34c-2c4c-4074-8761-e162f5d2f138 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:43.326739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1364/2716] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:43.331478 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f064353a-69ab-4d13-a308-5fc4214045dc admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:43.342027 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f064353a-69ab-4d13-a308-5fc4214045dc admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:43.347578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e73ce7-ca93-497a-9e8e-d13780d6851f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:43.348179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e73ce7-ca93-497a-9e8e-d13780d6851f admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.348667 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e73ce7-ca93-497a-9e8e-d13780d6851f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.348946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e73ce7-ca93-497a-9e8e-d13780d6851f admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.349596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e73ce7-ca93-497a-9e8e-d13780d6851f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:43.351192 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e73ce7-ca93-497a-9e8e-d13780d6851f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:43.365599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1353/2717] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:43.367400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28afc34c-2c4c-4074-8761-e162f5d2f138 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:43.378082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28afc34c-2c4c-4074-8761-e162f5d2f138 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:43.382926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4909e775-ad52-45ea-b5d7-f892cd56ba8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:43.383632 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4909e775-ad52-45ea-b5d7-f892cd56ba8d admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.383915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4909e775-ad52-45ea-b5d7-f892cd56ba8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.384192 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4909e775-ad52-45ea-b5d7-f892cd56ba8d admin admin] PATH_INFO: `/v3/projects/4cb3a1186b384dbcb9e9e38cfda77471/users/2907b6ca671e4867b22a284c2e6a6284/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.400258 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4909e775-ad52-45ea-b5d7-f892cd56ba8d admin admin] RBAC: Authorizing `identity:create_grant(project_id=4cb3a1186b384dbcb9e9e38cfda77471, user_id=2907b6ca671e4867b22a284c2e6a6284, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:43.402992 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4909e775-ad52-45ea-b5d7-f892cd56ba8d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:43.416269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1365/2718] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:43 2022] PUT /identity/v3/projects/4cb3a1186b384dbcb9e9e38cfda77471/users/2907b6ca671e4867b22a284c2e6a6284/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:43.428597 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e353f0-9291-4ce8-a7f1-b73aa9d4c80f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.428883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e353f0-9291-4ce8-a7f1-b73aa9d4c80f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.429154 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e353f0-9291-4ce8-a7f1-b73aa9d4c80f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.458036 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a1e353f0-9291-4ce8-a7f1-b73aa9d4c80f None None] MFA Rules not processed for user `2907b6ca671e4867b22a284c2e6a6284`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:43.459232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a1e353f0-9291-4ce8-a7f1-b73aa9d4c80f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:43.523364 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1354/2719] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:43.535356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4909e775-ad52-45ea-b5d7-f892cd56ba8d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:43.537158 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4909e775-ad52-45ea-b5d7-f892cd56ba8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:43.598288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4909e775-ad52-45ea-b5d7-f892cd56ba8d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:43.602047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f081982f-2fd1-4482-a9e5-c9f562be0a6f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:43.602652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f081982f-2fd1-4482-a9e5-c9f562be0a6f service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.602933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f081982f-2fd1-4482-a9e5-c9f562be0a6f service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.603210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f081982f-2fd1-4482-a9e5-c9f562be0a6f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.612939 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f081982f-2fd1-4482-a9e5-c9f562be0a6f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:43.614453 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f081982f-2fd1-4482-a9e5-c9f562be0a6f service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:43.625057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1366/2720] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:43 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:43.653574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e353f0-9291-4ce8-a7f1-b73aa9d4c80f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:43.713529 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:43.728438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e353f0-9291-4ce8-a7f1-b73aa9d4c80f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:43.736081 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:43.736793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.737085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.737361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] PATH_INFO: `/v3/users/2907b6ca671e4867b22a284c2e6a6284` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.738686 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2907b6ca671e4867b22a284c2e6a6284)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:43.740728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:43.800610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2907b6ca671e4867b22a284c2e6a6284'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:43.813489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2907b6ca671e4867b22a284c2e6a6284'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:43.814825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2907b6ca671e4867b22a284c2e6a6284'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:43.815751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2907b6ca671e4867b22a284c2e6a6284'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:43.823852 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2907b6ca671e4867b22a284c2e6a6284'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:43.842728 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1355/2721] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:43:43 2022] DELETE /identity/v3/users/2907b6ca671e4867b22a284c2e6a6284 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:43.848475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccd8110-2577-49ed-8262-aa0a274ecb4a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:43.848787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccd8110-2577-49ed-8262-aa0a274ecb4a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:43.849043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccd8110-2577-49ed-8262-aa0a274ecb4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:43.869225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-0ccd8110-2577-49ed-8262-aa0a274ecb4a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:43.870460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0ccd8110-2577-49ed-8262-aa0a274ecb4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:43.910472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:43.938099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1367/2722] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:43.948191 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:43.949771 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:44.013366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07e7e31-3d9b-43ea-ab50-28265795abc5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:44.017314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23efdcc8-a308-41fc-a14e-23f746ec315e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:44.017914 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23efdcc8-a308-41fc-a14e-23f746ec315e service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.018197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23efdcc8-a308-41fc-a14e-23f746ec315e service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.018473 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23efdcc8-a308-41fc-a14e-23f746ec315e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.027916 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23efdcc8-a308-41fc-a14e-23f746ec315e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:44.029384 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23efdcc8-a308-41fc-a14e-23f746ec315e service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:44.042538 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1356/2723] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:44.318944 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccd8110-2577-49ed-8262-aa0a274ecb4a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:44.321064 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-0ccd8110-2577-49ed-8262-aa0a274ecb4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:44.335213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccd8110-2577-49ed-8262-aa0a274ecb4a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:44.341323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:44.341962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.342246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.342523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] PATH_INFO: `/v3/projects/4cb3a1186b384dbcb9e9e38cfda77471` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.344175 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] RBAC: Authorizing `identity:delete_project(project_id=4cb3a1186b384dbcb9e9e38cfda77471)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:44.347079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:44.399608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4cb3a1186b384dbcb9e9e38cfda77471', 'request_id': 'req-7422a2fe-095a-4bba-80b6-f3e645f42e5e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:44.406981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4cb3a1186b384dbcb9e9e38cfda77471', 'request_id': 'req-7422a2fe-095a-4bba-80b6-f3e645f42e5e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:44.413814 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4cb3a1186b384dbcb9e9e38cfda77471', 'request_id': 'req-7422a2fe-095a-4bba-80b6-f3e645f42e5e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:44.429653 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] The token cache is being invalidate because project 4cb3a1186b384dbcb9e9e38cfda77471 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:43:44.430102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:44.430786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-7422a2fe-095a-4bba-80b6-f3e645f42e5e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:43:44.432374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1368/2724] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:43:44 2022] DELETE /identity/v3/projects/4cb3a1186b384dbcb9e9e38cfda77471 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:44.438475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3494ca86-6716-4671-9b73-32511088a978 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.439187 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3494ca86-6716-4671-9b73-32511088a978 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.439285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3494ca86-6716-4671-9b73-32511088a978 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.460153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3494ca86-6716-4671-9b73-32511088a978 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:44.461417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3494ca86-6716-4671-9b73-32511088a978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:44.531393 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1357/2725] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:44.539378 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0597a0-4f7e-4f9f-b4e2-e60d533434d3 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.539658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0597a0-4f7e-4f9f-b4e2-e60d533434d3 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.539930 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0597a0-4f7e-4f9f-b4e2-e60d533434d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.560318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-aa0597a0-4f7e-4f9f-b4e2-e60d533434d3 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:44.561574 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-aa0597a0-4f7e-4f9f-b4e2-e60d533434d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:44.571966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1369/2726] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:43:44.576133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3494ca86-6716-4671-9b73-32511088a978 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:44.586800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3494ca86-6716-4671-9b73-32511088a978 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:44.591606 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4235d0ef-898b-407e-a919-fdb78ca21d7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:44.592200 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4235d0ef-898b-407e-a919-fdb78ca21d7a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.592515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4235d0ef-898b-407e-a919-fdb78ca21d7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.592810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4235d0ef-898b-407e-a919-fdb78ca21d7a admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.593466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4235d0ef-898b-407e-a919-fdb78ca21d7a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:43:44.593777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4235d0ef-898b-407e-a919-fdb78ca21d7a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:44.595413 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4235d0ef-898b-407e-a919-fdb78ca21d7a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:44.607703 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1358/2727] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:43:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:43:44.612418 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0597a0-4f7e-4f9f-b4e2-e60d533434d3 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:44.622001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0597a0-4f7e-4f9f-b4e2-e60d533434d3 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:44.627773 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098c4505-8671-4f12-8691-81d811fcba40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:44.628406 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098c4505-8671-4f12-8691-81d811fcba40 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.628684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098c4505-8671-4f12-8691-81d811fcba40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.628953 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098c4505-8671-4f12-8691-81d811fcba40 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.629587 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098c4505-8671-4f12-8691-81d811fcba40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:44.631489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098c4505-8671-4f12-8691-81d811fcba40 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:44.664966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1370/2728] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:44 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:44.666334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4235d0ef-898b-407e-a919-fdb78ca21d7a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:44.673793 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:44.677649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4235d0ef-898b-407e-a919-fdb78ca21d7a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:44.742778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10180559-f115-460b-8490-2f2e91d9f661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:44.743291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10180559-f115-460b-8490-2f2e91d9f661 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.743571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10180559-f115-460b-8490-2f2e91d9f661 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.743845 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10180559-f115-460b-8490-2f2e91d9f661 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.744545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10180559-f115-460b-8490-2f2e91d9f661 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:44.746500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10180559-f115-460b-8490-2f2e91d9f661 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:44.774133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1359/2729] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:44 2022] POST /identity/v3/users => generated 293 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:44.775740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098c4505-8671-4f12-8691-81d811fcba40 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:44.789115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098c4505-8671-4f12-8691-81d811fcba40 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:44.793860 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876d311-8dde-4d15-afce-fdbd9088bc30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:44.794471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876d311-8dde-4d15-afce-fdbd9088bc30 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.794746 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876d311-8dde-4d15-afce-fdbd9088bc30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.795015 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876d311-8dde-4d15-afce-fdbd9088bc30 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.795658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8876d311-8dde-4d15-afce-fdbd9088bc30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:44.798210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8876d311-8dde-4d15-afce-fdbd9088bc30 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:44.810128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1371/2730] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:44 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:44.814955 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10180559-f115-460b-8490-2f2e91d9f661 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:44.825640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10180559-f115-460b-8490-2f2e91d9f661 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:44.831197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52beb85b-577a-44b4-a934-ec4364557e00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:44.831811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52beb85b-577a-44b4-a934-ec4364557e00 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.832090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52beb85b-577a-44b4-a934-ec4364557e00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.832389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52beb85b-577a-44b4-a934-ec4364557e00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.833046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52beb85b-577a-44b4-a934-ec4364557e00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:44.834668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52beb85b-577a-44b4-a934-ec4364557e00 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:44.847933 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1360/2731] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:44 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:44.852197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876d311-8dde-4d15-afce-fdbd9088bc30 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:44.863466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876d311-8dde-4d15-afce-fdbd9088bc30 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:44.868354 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0abf15-091d-4c41-83c9-4b259b22e5d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:44.869164 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0abf15-091d-4c41-83c9-4b259b22e5d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.869530 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0abf15-091d-4c41-83c9-4b259b22e5d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.869985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0abf15-091d-4c41-83c9-4b259b22e5d3 admin admin] PATH_INFO: `/v3/projects/63b4276ece354a06b400b45e9e49186b/users/e1d3321cdb59406da03c91fd8ec71885/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.886649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0abf15-091d-4c41-83c9-4b259b22e5d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63b4276ece354a06b400b45e9e49186b, user_id=e1d3321cdb59406da03c91fd8ec71885, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:44.889438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0abf15-091d-4c41-83c9-4b259b22e5d3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:44.904795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1372/2732] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:44 2022] PUT /identity/v3/projects/63b4276ece354a06b400b45e9e49186b/users/e1d3321cdb59406da03c91fd8ec71885/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:44.910383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415a2b2-218e-4ae4-be7f-36f2d89683f0 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:44.910677 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415a2b2-218e-4ae4-be7f-36f2d89683f0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:44.910950 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d415a2b2-218e-4ae4-be7f-36f2d89683f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:44.930184 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d415a2b2-218e-4ae4-be7f-36f2d89683f0 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:44.931675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d415a2b2-218e-4ae4-be7f-36f2d89683f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:44.998355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1361/2733] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:45.008758 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0abf15-091d-4c41-83c9-4b259b22e5d3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:45.010475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9d0abf15-091d-4c41-83c9-4b259b22e5d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:45.075000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0abf15-091d-4c41-83c9-4b259b22e5d3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:45.079159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7977f45c-70fb-437e-b90b-ad04e861b0d3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:45.079759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7977f45c-70fb-437e-b90b-ad04e861b0d3 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:45.080035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7977f45c-70fb-437e-b90b-ad04e861b0d3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:45.080303 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7977f45c-70fb-437e-b90b-ad04e861b0d3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:45.089646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7977f45c-70fb-437e-b90b-ad04e861b0d3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:45.091067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7977f45c-70fb-437e-b90b-ad04e861b0d3 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:45.101752 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1373/2734] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:49.159966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d65997-58ef-4ec9-93e5-593ac8e2fb68 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:49.160568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d65997-58ef-4ec9-93e5-593ac8e2fb68 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:49.160694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d65997-58ef-4ec9-93e5-593ac8e2fb68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:49.211388 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-77d65997-58ef-4ec9-93e5-593ac8e2fb68 None None] MFA Rules not processed for user `e1d3321cdb59406da03c91fd8ec71885`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:49.212873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-77d65997-58ef-4ec9-93e5-593ac8e2fb68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:49.281896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1362/2735] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:49 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:49.286501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7977f45c-70fb-437e-b90b-ad04e861b0d3 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:49.291441 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:49.298273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7977f45c-70fb-437e-b90b-ad04e861b0d3 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:49.304285 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a146676-8c25-4b13-9ce3-0c4293317d96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:49.304921 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a146676-8c25-4b13-9ce3-0c4293317d96 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:49.305271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a146676-8c25-4b13-9ce3-0c4293317d96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:49.305554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a146676-8c25-4b13-9ce3-0c4293317d96 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:49.306184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a146676-8c25-4b13-9ce3-0c4293317d96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:49.308115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a146676-8c25-4b13-9ce3-0c4293317d96 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:49.337803 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1374/2736] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:43:49 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:43:49.341849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d65997-58ef-4ec9-93e5-593ac8e2fb68 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:49.353246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d65997-58ef-4ec9-93e5-593ac8e2fb68 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:49.364125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:49.430521 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5700d0-b886-4e00-aa23-c5faf32260d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:49.431133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5700d0-b886-4e00-aa23-c5faf32260d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:49.431449 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5700d0-b886-4e00-aa23-c5faf32260d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:49.431727 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5700d0-b886-4e00-aa23-c5faf32260d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:49.432369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5700d0-b886-4e00-aa23-c5faf32260d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:49.434331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5700d0-b886-4e00-aa23-c5faf32260d6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:49.465484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1363/2737] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:43:49 2022] POST /identity/v3/users => generated 299 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:43:49.469730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a146676-8c25-4b13-9ce3-0c4293317d96 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:49.481721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a146676-8c25-4b13-9ce3-0c4293317d96 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:49.487795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149ce180-8c8b-4e77-b81d-0dfb3a2b11ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:49.488853 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149ce180-8c8b-4e77-b81d-0dfb3a2b11ff admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:49.489290 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149ce180-8c8b-4e77-b81d-0dfb3a2b11ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:49.489756 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149ce180-8c8b-4e77-b81d-0dfb3a2b11ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:49.490619 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149ce180-8c8b-4e77-b81d-0dfb3a2b11ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:49.492534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149ce180-8c8b-4e77-b81d-0dfb3a2b11ff admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:49.505436 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1375/2738] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:49 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:43:49.509576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5700d0-b886-4e00-aa23-c5faf32260d6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:49.520177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5700d0-b886-4e00-aa23-c5faf32260d6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:49.525500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f52f17-9281-4146-895a-6aeaf9ad7e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:49.526178 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f52f17-9281-4146-895a-6aeaf9ad7e32 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:49.526462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f52f17-9281-4146-895a-6aeaf9ad7e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:49.527559 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f52f17-9281-4146-895a-6aeaf9ad7e32 admin admin] PATH_INFO: `/v3/projects/18dcc556be884d28bd820d0b7bbb493e/users/82122b4607ed410196874b7413a7b4ad/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:49.547306 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f52f17-9281-4146-895a-6aeaf9ad7e32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18dcc556be884d28bd820d0b7bbb493e, user_id=82122b4607ed410196874b7413a7b4ad, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:49.550890 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f52f17-9281-4146-895a-6aeaf9ad7e32 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:49.571248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1364/2739] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:49 2022] PUT /identity/v3/projects/18dcc556be884d28bd820d0b7bbb493e/users/82122b4607ed410196874b7413a7b4ad/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:43:49.572633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149ce180-8c8b-4e77-b81d-0dfb3a2b11ff admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:49.584708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149ce180-8c8b-4e77-b81d-0dfb3a2b11ff admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:49.656288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d75ecf-e587-4b84-91b8-fc0ac44a2a42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:49.656984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d75ecf-e587-4b84-91b8-fc0ac44a2a42 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:49.657262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d75ecf-e587-4b84-91b8-fc0ac44a2a42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:49.657534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d75ecf-e587-4b84-91b8-fc0ac44a2a42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:49.658185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d75ecf-e587-4b84-91b8-fc0ac44a2a42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:49.659806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d75ecf-e587-4b84-91b8-fc0ac44a2a42 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:49.670266 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1376/2740] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:43:49 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:43:49.675789 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f52f17-9281-4146-895a-6aeaf9ad7e32 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:49.705433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f52f17-9281-4146-895a-6aeaf9ad7e32 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:49.711334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474bc8b0-c4df-4faa-a735-eaf6eba30548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:49.712096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474bc8b0-c4df-4faa-a735-eaf6eba30548 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:49.712585 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474bc8b0-c4df-4faa-a735-eaf6eba30548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:49.712954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474bc8b0-c4df-4faa-a735-eaf6eba30548 admin admin] PATH_INFO: `/v3/projects/18dcc556be884d28bd820d0b7bbb493e/users/82122b4607ed410196874b7413a7b4ad/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:49.714970 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474bc8b0-c4df-4faa-a735-eaf6eba30548 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18dcc556be884d28bd820d0b7bbb493e, user_id=82122b4607ed410196874b7413a7b4ad, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:49.718334 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474bc8b0-c4df-4faa-a735-eaf6eba30548 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:49.729678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1365/2741] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:43:49 2022] PUT /identity/v3/projects/18dcc556be884d28bd820d0b7bbb493e/users/82122b4607ed410196874b7413a7b4ad/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:43:54.466839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fa6f2b-0b9b-4184-bde4-eeec69771498 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:54.467984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fa6f2b-0b9b-4184-bde4-eeec69771498 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:54.468706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fa6f2b-0b9b-4184-bde4-eeec69771498 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:54.489640 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-d5fa6f2b-0b9b-4184-bde4-eeec69771498 None None] MFA Rules not processed for user `a47ec612bce54bc8b2062b03d36d10d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:54.491179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d5fa6f2b-0b9b-4184-bde4-eeec69771498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:54.582218 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1377/2742] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:54 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:55.939112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a84427-5909-46cc-8a98-dac62a3e81c9 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:55.939548 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a84427-5909-46cc-8a98-dac62a3e81c9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:55.939695 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a84427-5909-46cc-8a98-dac62a3e81c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:56.015599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-78a84427-5909-46cc-8a98-dac62a3e81c9 None None] MFA Rules not processed for user `82122b4607ed410196874b7413a7b4ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:56.016873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-78a84427-5909-46cc-8a98-dac62a3e81c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:56.132095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1366/2743] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:43:55 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:56.147222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fa6f2b-0b9b-4184-bde4-eeec69771498 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:56.149388 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d5fa6f2b-0b9b-4184-bde4-eeec69771498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:56.235432 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9dbc75-d264-4fa4-9f26-557e8ddb4cb7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:56.235717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9dbc75-d264-4fa4-9f26-557e8ddb4cb7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:56.236044 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9dbc75-d264-4fa4-9f26-557e8ddb4cb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:56.266390 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ea9dbc75-d264-4fa4-9f26-557e8ddb4cb7 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:56.268054 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ea9dbc75-d264-4fa4-9f26-557e8ddb4cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:56.270952 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fa6f2b-0b9b-4184-bde4-eeec69771498 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:56.290703 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab989b69-0488-4c86-a73a-d383434cea16 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e07b6890d3004dd78faed48a1ab90086', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:56.293227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab989b69-0488-4c86-a73a-d383434cea16 service glance] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:56.294268 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab989b69-0488-4c86-a73a-d383434cea16 service glance] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:56.294682 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab989b69-0488-4c86-a73a-d383434cea16 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:56.357882 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:56.361422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:43:56.377360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1367/2744] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:43:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:56.477173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab989b69-0488-4c86-a73a-d383434cea16 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:56.478651 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab989b69-0488-4c86-a73a-d383434cea16 service glance] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:56.483810 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9dbc75-d264-4fa4-9f26-557e8ddb4cb7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:56.486459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ea9dbc75-d264-4fa4-9f26-557e8ddb4cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:56.511608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1378/2745] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:43:56 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:43:56.591485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9dbc75-d264-4fa4-9f26-557e8ddb4cb7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:56.600078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629a82fd-0823-48ac-8272-aae2e972e9e5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:56.600839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629a82fd-0823-48ac-8272-aae2e972e9e5 service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:56.601207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629a82fd-0823-48ac-8272-aae2e972e9e5 service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:56.601560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629a82fd-0823-48ac-8272-aae2e972e9e5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:56.627685 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629a82fd-0823-48ac-8272-aae2e972e9e5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:56.631774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629a82fd-0823-48ac-8272-aae2e972e9e5 service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:56.654717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1368/2746] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:43:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 26 01:43:56.798084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ba2b29-a823-4a84-9d73-4f76592edc88 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:56.798477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ba2b29-a823-4a84-9d73-4f76592edc88 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:56.798827 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ba2b29-a823-4a84-9d73-4f76592edc88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:56.836479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-10ba2b29-a823-4a84-9d73-4f76592edc88 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:56.837824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-10ba2b29-a823-4a84-9d73-4f76592edc88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:56.848721 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1379/2747] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:43:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:56.865383 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629a82fd-0823-48ac-8272-aae2e972e9e5 service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:56.886602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629a82fd-0823-48ac-8272-aae2e972e9e5 service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:56.896510 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bd4e03-3e18-49f2-91aa-51c9b860d0e2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:56.897277 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd4e03-3e18-49f2-91aa-51c9b860d0e2 service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:56.897557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd4e03-3e18-49f2-91aa-51c9b860d0e2 service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:56.897827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd4e03-3e18-49f2-91aa-51c9b860d0e2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:56.916095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bd4e03-3e18-49f2-91aa-51c9b860d0e2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:56.918545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bd4e03-3e18-49f2-91aa-51c9b860d0e2 service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:56.950476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1369/2748] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:43:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:43:57.750932 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f4a18f-7ec6-4ac6-92b0-bb57062e55fd None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:57.751840 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f4a18f-7ec6-4ac6-92b0-bb57062e55fd None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:57.752315 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f4a18f-7ec6-4ac6-92b0-bb57062e55fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:57.796246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e5f4a18f-7ec6-4ac6-92b0-bb57062e55fd None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:57.798000 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e5f4a18f-7ec6-4ac6-92b0-bb57062e55fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:57.807966 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1380/2749] 10.222.0.25 () {58 vars in 967 bytes} [Mon Sep 26 01:43:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:57.834932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bd4e03-3e18-49f2-91aa-51c9b860d0e2 service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:57.846959 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bd4e03-3e18-49f2-91aa-51c9b860d0e2 service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:57.855452 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af8e669-b22c-4f17-8803-651b7b0c25a9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:57.856500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af8e669-b22c-4f17-8803-651b7b0c25a9 service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:57.857041 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af8e669-b22c-4f17-8803-651b7b0c25a9 service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:57.857544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af8e669-b22c-4f17-8803-651b7b0c25a9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:57.872463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af8e669-b22c-4f17-8803-651b7b0c25a9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:57.875257 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af8e669-b22c-4f17-8803-651b7b0c25a9 service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:57.888619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1370/2750] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:43:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:43:58.324458 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d605a4-7dbd-4e2c-9589-d2a1e39b33e4 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:58.324997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d605a4-7dbd-4e2c-9589-d2a1e39b33e4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:58.325474 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d605a4-7dbd-4e2c-9589-d2a1e39b33e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:58.358950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-65d605a4-7dbd-4e2c-9589-d2a1e39b33e4 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:58.361093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-65d605a4-7dbd-4e2c-9589-d2a1e39b33e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:58.378027 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1381/2751] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:43:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:58.398952 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af8e669-b22c-4f17-8803-651b7b0c25a9 service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:58.417090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af8e669-b22c-4f17-8803-651b7b0c25a9 service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:58.426918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a7c529-0f9a-426b-bcf5-4eb856472cb7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:58.427760 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a7c529-0f9a-426b-bcf5-4eb856472cb7 service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:58.428040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a7c529-0f9a-426b-bcf5-4eb856472cb7 service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:58.428316 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a7c529-0f9a-426b-bcf5-4eb856472cb7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:58.440381 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a7c529-0f9a-426b-bcf5-4eb856472cb7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:58.442495 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a7c529-0f9a-426b-bcf5-4eb856472cb7 service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:58.452670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1371/2752] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:43:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:43:59.145395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bafbf1-9fd0-4aea-994c-5d5dc77c1030 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:59.146373 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bafbf1-9fd0-4aea-994c-5d5dc77c1030 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:59.147734 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bafbf1-9fd0-4aea-994c-5d5dc77c1030 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:59.179858 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e9bafbf1-9fd0-4aea-994c-5d5dc77c1030 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:43:59.182375 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e9bafbf1-9fd0-4aea-994c-5d5dc77c1030 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:43:59.207970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1382/2753] 10.222.0.25 () {58 vars in 967 bytes} [Mon Sep 26 01:43:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:43:59.244179 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a7c529-0f9a-426b-bcf5-4eb856472cb7 service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:43:59.258706 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a7c529-0f9a-426b-bcf5-4eb856472cb7 service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:43:59.270604 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9eb165-9782-401e-bd67-6c02a6fabe35 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:43:59.274011 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9eb165-9782-401e-bd67-6c02a6fabe35 service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:43:59.274939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9eb165-9782-401e-bd67-6c02a6fabe35 service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:43:59.275466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9eb165-9782-401e-bd67-6c02a6fabe35 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:43:59.289848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9eb165-9782-401e-bd67-6c02a6fabe35 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:43:59.292560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9eb165-9782-401e-bd67-6c02a6fabe35 service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:43:59.312776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1372/2754] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:43:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 26 01:44:02.176713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bafbf1-9fd0-4aea-994c-5d5dc77c1030 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:02.178299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e9bafbf1-9fd0-4aea-994c-5d5dc77c1030 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:02.300601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bafbf1-9fd0-4aea-994c-5d5dc77c1030 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:02.306970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:02.308329 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:02.308837 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:02.309489 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] PATH_INFO: `/v3/users/21d71eca7714406d8f97c061d6b26ce9` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:02.311577 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21d71eca7714406d8f97c061d6b26ce9)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:02.317102 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:02.405237 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21d71eca7714406d8f97c061d6b26ce9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:02.422053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21d71eca7714406d8f97c061d6b26ce9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:02.422984 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21d71eca7714406d8f97c061d6b26ce9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:02.424595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21d71eca7714406d8f97c061d6b26ce9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:02.444200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21d71eca7714406d8f97c061d6b26ce9'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:02.464863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1383/2755] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:44:02 2022] DELETE /identity/v3/users/21d71eca7714406d8f97c061d6b26ce9 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:44:02.469605 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1fc5dd-7a19-4903-89b2-be4ad87ff4c9 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:02.470145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1fc5dd-7a19-4903-89b2-be4ad87ff4c9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:02.470145 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1fc5dd-7a19-4903-89b2-be4ad87ff4c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:02.504320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9f1fc5dd-7a19-4903-89b2-be4ad87ff4c9 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:44:02.506232 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9f1fc5dd-7a19-4903-89b2-be4ad87ff4c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:02.611346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1373/2756] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:44:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:44:02.627398 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:02.630961 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:02.655151 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:44:02.722794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3894d2f-5ded-41dd-ab03-c97eb38a88b8 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:02.727324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf20cc9-93c9-42b1-afc4-c6d0f0f80115 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:02.728224 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf20cc9-93c9-42b1-afc4-c6d0f0f80115 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:02.728560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf20cc9-93c9-42b1-afc4-c6d0f0f80115 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:02.728832 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf20cc9-93c9-42b1-afc4-c6d0f0f80115 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:02.740526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf20cc9-93c9-42b1-afc4-c6d0f0f80115 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:02.741556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf20cc9-93c9-42b1-afc4-c6d0f0f80115 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:02.753131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1384/2757] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:44:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:44:02.965247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1fc5dd-7a19-4903-89b2-be4ad87ff4c9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:02.966651 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9f1fc5dd-7a19-4903-89b2-be4ad87ff4c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:02.986801 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1fc5dd-7a19-4903-89b2-be4ad87ff4c9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:03.005749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:03.007536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:03.008037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:03.008860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] PATH_INFO: `/v3/projects/10895902ec774f7ab15db01ce507471d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:03.016859 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=10895902ec774f7ab15db01ce507471d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:03.021463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:03.080885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:44:03.111315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '10895902ec774f7ab15db01ce507471d', 'request_id': 'req-fd66c169-64b0-48d3-986a-ec5f24e862b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:03.121247 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '10895902ec774f7ab15db01ce507471d', 'request_id': 'req-fd66c169-64b0-48d3-986a-ec5f24e862b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:03.133821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '10895902ec774f7ab15db01ce507471d', 'request_id': 'req-fd66c169-64b0-48d3-986a-ec5f24e862b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:03.147401 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] The token cache is being invalidate because project 10895902ec774f7ab15db01ce507471d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:44:03.147778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:03.148499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd66c169-64b0-48d3-986a-ec5f24e862b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:03.150580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1374/2758] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:44:02 2022] DELETE /identity/v3/projects/10895902ec774f7ab15db01ce507471d => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:44:08.453167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c20b2d-9be9-4a89-a992-c649270bf371 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:08.453689 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c20b2d-9be9-4a89-a992-c649270bf371 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:08.453796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c20b2d-9be9-4a89-a992-c649270bf371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:08.482807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-b1c20b2d-9be9-4a89-a992-c649270bf371 None None] MFA Rules not processed for user `e1d3321cdb59406da03c91fd8ec71885`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:44:08.483842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b1c20b2d-9be9-4a89-a992-c649270bf371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:08.559584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1385/2759] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:44:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:44:10.660149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d77d36-0118-4352-8542-0c7e6d39f748 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:10.660699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d77d36-0118-4352-8542-0c7e6d39f748 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:10.660800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d77d36-0118-4352-8542-0c7e6d39f748 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:10.691238 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-57d77d36-0118-4352-8542-0c7e6d39f748 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:44:10.692507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-57d77d36-0118-4352-8542-0c7e6d39f748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:10.792776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1375/2760] 10.222.0.25 () {58 vars in 1009 bytes} [Mon Sep 26 01:44:10 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:44:10.808732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c20b2d-9be9-4a89-a992-c649270bf371 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:10.810884 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b1c20b2d-9be9-4a89-a992-c649270bf371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:10.914146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c20b2d-9be9-4a89-a992-c649270bf371 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:10.922381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0da8ea-eead-4f56-a2be-df0a25954479 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:10.923212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0da8ea-eead-4f56-a2be-df0a25954479 service swift] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:10.923649 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0da8ea-eead-4f56-a2be-df0a25954479 service swift] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:10.923928 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0da8ea-eead-4f56-a2be-df0a25954479 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:10.937825 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0da8ea-eead-4f56-a2be-df0a25954479 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:10.940384 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0da8ea-eead-4f56-a2be-df0a25954479 service swift] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:10.953381 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1386/2761] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:44:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 26 01:44:17.176231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d77d36-0118-4352-8542-0c7e6d39f748 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:17.177719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-57d77d36-0118-4352-8542-0c7e6d39f748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:17.241920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d77d36-0118-4352-8542-0c7e6d39f748 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:17.245924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16a9111-1c11-4792-9a0a-5ea1d1a278c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:17.246534 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16a9111-1c11-4792-9a0a-5ea1d1a278c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:17.246937 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16a9111-1c11-4792-9a0a-5ea1d1a278c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:17.247107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16a9111-1c11-4792-9a0a-5ea1d1a278c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:17.253665 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a16a9111-1c11-4792-9a0a-5ea1d1a278c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:17.304576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:44:17.324913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16a9111-1c11-4792-9a0a-5ea1d1a278c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:17.326635 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16a9111-1c11-4792-9a0a-5ea1d1a278c1 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:17.336246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1376/2762] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:44:17 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:44:20.948202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:44:22.907609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0da8ea-eead-4f56-a2be-df0a25954479 service swift] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:22.910132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7c0da8ea-eead-4f56-a2be-df0a25954479 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:23.022456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0da8ea-eead-4f56-a2be-df0a25954479 service swift] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:23.028787 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f43687c-f70d-418e-8d4c-d58050b8492a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:23.029473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f43687c-f70d-418e-8d4c-d58050b8492a service placement] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:23.030259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f43687c-f70d-418e-8d4c-d58050b8492a service placement] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:23.030259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f43687c-f70d-418e-8d4c-d58050b8492a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:23.032017 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6f43687c-f70d-418e-8d4c-d58050b8492a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:23.051637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f43687c-f70d-418e-8d4c-d58050b8492a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:23.053436 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f43687c-f70d-418e-8d4c-d58050b8492a service placement] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:23.064702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1387/2763] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:44:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:44:27.331596 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=8 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:44:27.634206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16a9111-1c11-4792-9a0a-5ea1d1a278c1 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:27.654946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16a9111-1c11-4792-9a0a-5ea1d1a278c1 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:27.658707 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55fc25f-68b3-44fd-b384-0592d7c1a335 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:27.659317 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55fc25f-68b3-44fd-b384-0592d7c1a335 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:27.659600 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55fc25f-68b3-44fd-b384-0592d7c1a335 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:27.659979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55fc25f-68b3-44fd-b384-0592d7c1a335 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:27.661699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a55fc25f-68b3-44fd-b384-0592d7c1a335 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:27.782176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55fc25f-68b3-44fd-b384-0592d7c1a335 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:27.784116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55fc25f-68b3-44fd-b384-0592d7c1a335 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:27.805768 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1377/2764] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:44:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:44:33.059732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=41 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:44:36.408910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f43687c-f70d-418e-8d4c-d58050b8492a service placement] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:36.411799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6f43687c-f70d-418e-8d4c-d58050b8492a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:36.431499 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f43687c-f70d-418e-8d4c-d58050b8492a service placement] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:36.437181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:36.438104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:36.438425 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:36.438780 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] PATH_INFO: `/v3/users/5c72b19a91804de398c476c5e1c404cf` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:36.440095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c72b19a91804de398c476c5e1c404cf)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:36.442253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:36.482074 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a40f99-a05d-4c28-9bdb-e08eeb89da81 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:36.482796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a40f99-a05d-4c28-9bdb-e08eeb89da81 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:36.482796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a40f99-a05d-4c28-9bdb-e08eeb89da81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:36.498970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c72b19a91804de398c476c5e1c404cf'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:36.502663 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-80a40f99-a05d-4c28-9bdb-e08eeb89da81 None None] MFA Rules not processed for user `64c98c1da1a847389fa035d86f6326de`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:44:36.504037 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-80a40f99-a05d-4c28-9bdb-e08eeb89da81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:36.507042 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c72b19a91804de398c476c5e1c404cf'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:36.507737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c72b19a91804de398c476c5e1c404cf'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:36.508412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c72b19a91804de398c476c5e1c404cf'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:36.520976 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c72b19a91804de398c476c5e1c404cf'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:36.533483 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1388/2765] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:44:36 2022] DELETE /identity/v3/users/5c72b19a91804de398c476c5e1c404cf => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:44:36.568457 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1378/2766] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:44:36 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:44:36.729546 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:36.731228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:36.856986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58693e0-ecc0-44f3-999e-81328783320d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:36.863393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:36.864182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:36.864510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:36.864823 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] PATH_INFO: `/v3/projects/d1d406b57f894053834ce17fa63fa6ac` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:36.866036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1d406b57f894053834ce17fa63fa6ac)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:36.868908 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:36.935810 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1d406b57f894053834ce17fa63fa6ac', 'request_id': 'req-6997fcd2-8db2-4c92-97e5-d4947fe1b102'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:36.943719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1d406b57f894053834ce17fa63fa6ac', 'request_id': 'req-6997fcd2-8db2-4c92-97e5-d4947fe1b102'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:36.952389 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1d406b57f894053834ce17fa63fa6ac', 'request_id': 'req-6997fcd2-8db2-4c92-97e5-d4947fe1b102'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:36.967045 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] The token cache is being invalidate because project d1d406b57f894053834ce17fa63fa6ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:44:36.967346 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:36.968009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:36.974167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1389/2767] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:44:36 2022] DELETE /identity/v3/projects/d1d406b57f894053834ce17fa63fa6ac => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:44:36.975678 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a40f99-a05d-4c28-9bdb-e08eeb89da81 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:36.992602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a40f99-a05d-4c28-9bdb-e08eeb89da81 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:37.073204 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:37.073854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:37.074792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:37.075046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] PATH_INFO: `/v3/users/a2d5286edc614fc0b69ac0e52879e426` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:37.077607 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2d5286edc614fc0b69ac0e52879e426)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:37.079954 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:37.083269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:44:37.143515 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2d5286edc614fc0b69ac0e52879e426'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:37.151679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2d5286edc614fc0b69ac0e52879e426'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:37.153318 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2d5286edc614fc0b69ac0e52879e426'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:37.154112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2d5286edc614fc0b69ac0e52879e426'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:37.164535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2d5286edc614fc0b69ac0e52879e426'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:37.201376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1379/2768] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:44:36 2022] DELETE /identity/v3/users/a2d5286edc614fc0b69ac0e52879e426 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:44:37.438916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:37.441487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:37.461625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:44:37.490307 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6997fcd2-8db2-4c92-97e5-d4947fe1b102 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:37.594223 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:37.594857 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:37.595977 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:37.596467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] PATH_INFO: `/v3/projects/020731480e384eb78efd7d5d71df6319` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:37.597927 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=020731480e384eb78efd7d5d71df6319)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:37.600057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:37.649188 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '020731480e384eb78efd7d5d71df6319', 'request_id': 'req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:37.655995 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '020731480e384eb78efd7d5d71df6319', 'request_id': 'req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:37.661507 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '020731480e384eb78efd7d5d71df6319', 'request_id': 'req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:37.675759 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] The token cache is being invalidate because project 020731480e384eb78efd7d5d71df6319 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:44:37.676053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:37.676700 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-afd1aa56-e5b4-4145-b6d4-1e06f6257a11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:44:37.678181 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1390/2769] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:44:37 2022] DELETE /identity/v3/projects/020731480e384eb78efd7d5d71df6319 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:44:44.113164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:44:44.114615 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:44.181857 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cb4a1d-a155-4e65-a63b-865386e47efe admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:44:44.228779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493823fc-0671-455f-ba0b-416481ce3459 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:44:44.229373 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493823fc-0671-455f-ba0b-416481ce3459 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:44:44.229647 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493823fc-0671-455f-ba0b-416481ce3459 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:44:44.230417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493823fc-0671-455f-ba0b-416481ce3459 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:44:44.232248 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-493823fc-0671-455f-ba0b-416481ce3459 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:44:44.271696 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:44:44.295250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493823fc-0671-455f-ba0b-416481ce3459 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:44:44.296791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493823fc-0671-455f-ba0b-416481ce3459 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:44:44.306471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1380/2770] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:44:44 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:44:47.672813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=50,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:44:54.302098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=11 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:45:43.220269 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a325ff-ed1c-4def-9b07-1573a3ed9095 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:45:43.220818 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a325ff-ed1c-4def-9b07-1573a3ed9095 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:45:43.220939 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a325ff-ed1c-4def-9b07-1573a3ed9095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:45:43.240022 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-e7a325ff-ed1c-4def-9b07-1573a3ed9095 None None] MFA Rules not processed for user `64c98c1da1a847389fa035d86f6326de`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:45:43.241274 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-e7a325ff-ed1c-4def-9b07-1573a3ed9095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:45:43.298361 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1391/2771] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:45:43 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:45:53.291161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=25 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:02.704215 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493823fc-0671-455f-ba0b-416481ce3459 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:02.705636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-493823fc-0671-455f-ba0b-416481ce3459 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:02.781472 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493823fc-0671-455f-ba0b-416481ce3459 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:02.785211 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4f7f7c-7d6e-461a-9580-7e5a40ec8792 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:02.785815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4f7f7c-7d6e-461a-9580-7e5a40ec8792 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:02.786091 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4f7f7c-7d6e-461a-9580-7e5a40ec8792 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:02.786362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4f7f7c-7d6e-461a-9580-7e5a40ec8792 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:02.788059 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9d4f7f7c-7d6e-461a-9580-7e5a40ec8792 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:02.846824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4f7f7c-7d6e-461a-9580-7e5a40ec8792 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:02.848243 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4f7f7c-7d6e-461a-9580-7e5a40ec8792 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:02.857201 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1381/2772] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:46:02 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:46:02.972104 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5869f77a-f6b8-4e2b-a1e9-3d9b1a288891 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:02.972631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5869f77a-f6b8-4e2b-a1e9-3d9b1a288891 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:02.972729 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5869f77a-f6b8-4e2b-a1e9-3d9b1a288891 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:02.992044 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-5869f77a-f6b8-4e2b-a1e9-3d9b1a288891 None None] MFA Rules not processed for user `a47ec612bce54bc8b2062b03d36d10d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:02.993311 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5869f77a-f6b8-4e2b-a1e9-3d9b1a288891 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:03.000371 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1392/2773] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:03.192662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4f7f7c-7d6e-461a-9580-7e5a40ec8792 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:03.194157 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9d4f7f7c-7d6e-461a-9580-7e5a40ec8792 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:03.207746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4f7f7c-7d6e-461a-9580-7e5a40ec8792 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:03.211462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82d79f9-817d-4f08-9981-79ff60a056f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:03.212063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82d79f9-817d-4f08-9981-79ff60a056f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:03.212429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82d79f9-817d-4f08-9981-79ff60a056f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:03.212652 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82d79f9-817d-4f08-9981-79ff60a056f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:03.214398 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b82d79f9-817d-4f08-9981-79ff60a056f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:03.300637 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82d79f9-817d-4f08-9981-79ff60a056f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:03.302080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82d79f9-817d-4f08-9981-79ff60a056f9 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:03.320223 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:03.324554 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1382/2774] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:46:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:46:04.897569 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5869f77a-f6b8-4e2b-a1e9-3d9b1a288891 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:04.899078 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5869f77a-f6b8-4e2b-a1e9-3d9b1a288891 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:04.911084 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5869f77a-f6b8-4e2b-a1e9-3d9b1a288891 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:04.914601 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d6bbb2-ca2e-45a8-a56b-7b5bf0544df6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:04.915215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d6bbb2-ca2e-45a8-a56b-7b5bf0544df6 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:04.915516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d6bbb2-ca2e-45a8-a56b-7b5bf0544df6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:04.915789 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d6bbb2-ca2e-45a8-a56b-7b5bf0544df6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:04.917465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c1d6bbb2-ca2e-45a8-a56b-7b5bf0544df6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:04.931109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d6bbb2-ca2e-45a8-a56b-7b5bf0544df6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:04.932556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d6bbb2-ca2e-45a8-a56b-7b5bf0544df6 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:04.944253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1393/2775] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:46:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:46:08.426850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3bee41-6235-46c8-a2e0-732f78acf5e4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:08.427327 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3bee41-6235-46c8-a2e0-732f78acf5e4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:08.427581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3bee41-6235-46c8-a2e0-732f78acf5e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:08.449331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0d3bee41-6235-46c8-a2e0-732f78acf5e4 None None] MFA Rules not processed for user `50a8d0782b6f47d1aec027c6e15018a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:08.451342 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0d3bee41-6235-46c8-a2e0-732f78acf5e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:08.535867 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1383/2776] 10.222.0.25 () {58 vars in 967 bytes} [Mon Sep 26 01:46:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:08.577631 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d6bbb2-ca2e-45a8-a56b-7b5bf0544df6 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:08.580106 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-c1d6bbb2-ca2e-45a8-a56b-7b5bf0544df6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:08.671299 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d6bbb2-ca2e-45a8-a56b-7b5bf0544df6 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:08.676274 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f5e425-e74d-4e4c-99f7-a443822e466e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e542b14838e2466fb59d17e9a20682dd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:08.676896 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f5e425-e74d-4e4c-99f7-a443822e466e service swift] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:08.677172 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f5e425-e74d-4e4c-99f7-a443822e466e service swift] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:08.677446 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f5e425-e74d-4e4c-99f7-a443822e466e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:08.685855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f5e425-e74d-4e4c-99f7-a443822e466e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:08.687560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f5e425-e74d-4e4c-99f7-a443822e466e service swift] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:08.695427 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1394/2777] 10.222.0.25 () {60 vars in 1384 bytes} [Mon Sep 26 01:46:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 26 01:46:15.586516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3bee41-6235-46c8-a2e0-732f78acf5e4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:15.588007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0d3bee41-6235-46c8-a2e0-732f78acf5e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:15.675018 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3bee41-6235-46c8-a2e0-732f78acf5e4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:15.679557 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:15.680181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:15.680500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:15.680776 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] PATH_INFO: `/v3/users/e1d3321cdb59406da03c91fd8ec71885` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:15.681915 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1d3321cdb59406da03c91fd8ec71885)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:15.683840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:15.729743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1d3321cdb59406da03c91fd8ec71885'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:15.736366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1d3321cdb59406da03c91fd8ec71885'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:15.737001 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1d3321cdb59406da03c91fd8ec71885'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:15.737610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1d3321cdb59406da03c91fd8ec71885'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:15.744581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1d3321cdb59406da03c91fd8ec71885'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:15.762206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1384/2778] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:46:15 2022] DELETE /identity/v3/users/e1d3321cdb59406da03c91fd8ec71885 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:15.949741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f5e425-e74d-4e4c-99f7-a443822e466e service swift] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:15.961669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f5e425-e74d-4e4c-99f7-a443822e466e service swift] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:16.053279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:16.053923 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:16.054201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:16.055179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] PATH_INFO: `/v3/projects/63b4276ece354a06b400b45e9e49186b` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:16.056630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] RBAC: Authorizing `identity:delete_project(project_id=63b4276ece354a06b400b45e9e49186b)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:16.058639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:16.108255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63b4276ece354a06b400b45e9e49186b', 'request_id': 'req-20e4895b-88df-4c90-9541-4e2e9358279a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.114225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63b4276ece354a06b400b45e9e49186b', 'request_id': 'req-20e4895b-88df-4c90-9541-4e2e9358279a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.119726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63b4276ece354a06b400b45e9e49186b', 'request_id': 'req-20e4895b-88df-4c90-9541-4e2e9358279a'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.130280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:16.135242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] The token cache is being invalidate because project 63b4276ece354a06b400b45e9e49186b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:46:16.135795 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.136556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.138536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1395/2779] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:46:15 2022] DELETE /identity/v3/projects/63b4276ece354a06b400b45e9e49186b => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:46:16.142679 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:16.144164 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:16.157295 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3057de0b-3acb-429b-b3a3-f27ff1d4075a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:16.177378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:16.218182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:16.218800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:16.219076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:16.219346 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] PATH_INFO: `/v3/users/82122b4607ed410196874b7413a7b4ad` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:16.220501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] RBAC: Authorizing `identity:delete_user(user_id=82122b4607ed410196874b7413a7b4ad)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:16.222422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:16.267815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82122b4607ed410196874b7413a7b4ad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.275720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82122b4607ed410196874b7413a7b4ad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.276370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82122b4607ed410196874b7413a7b4ad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.277080 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82122b4607ed410196874b7413a7b4ad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.284535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82122b4607ed410196874b7413a7b4ad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.301341 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1385/2780] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:46:16 2022] DELETE /identity/v3/users/82122b4607ed410196874b7413a7b4ad => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:16.478203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:16.479726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:16.499008 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e4895b-88df-4c90-9541-4e2e9358279a admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:16.556824 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:16.557450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:16.557726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:16.557993 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] PATH_INFO: `/v3/projects/18dcc556be884d28bd820d0b7bbb493e` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:16.559095 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=18dcc556be884d28bd820d0b7bbb493e)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:16.562321 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:16.607788 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18dcc556be884d28bd820d0b7bbb493e', 'request_id': 'req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.613956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18dcc556be884d28bd820d0b7bbb493e', 'request_id': 'req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.620089 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18dcc556be884d28bd820d0b7bbb493e', 'request_id': 'req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.634362 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] The token cache is being invalidate because project 18dcc556be884d28bd820d0b7bbb493e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:46:16.634652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.635287 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:16.636829 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1396/2781] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:46:16 2022] DELETE /identity/v3/projects/18dcc556be884d28bd820d0b7bbb493e => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:26.297827 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=42,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:26.630956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=51,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:52.588835 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:52.590443 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:52.606981 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5134db7d-218a-4e31-bff5-f72009da776e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:52.668058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:52.668815 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:52.669098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:52.669372 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] PATH_INFO: `/v3/users/64c98c1da1a847389fa035d86f6326de` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:52.670619 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] RBAC: Authorizing `identity:delete_user(user_id=64c98c1da1a847389fa035d86f6326de)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:52.672623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:52.721722 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '64c98c1da1a847389fa035d86f6326de'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:52.728929 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '64c98c1da1a847389fa035d86f6326de'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:52.729744 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '64c98c1da1a847389fa035d86f6326de'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:52.730550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '64c98c1da1a847389fa035d86f6326de'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:52.738563 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-704c7559-b604-4d3d-a7a4-4d273a5d9039 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '64c98c1da1a847389fa035d86f6326de'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:52.756163 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1386/2782] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:46:52 2022] DELETE /identity/v3/users/64c98c1da1a847389fa035d86f6326de => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:52.931313 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:52.933504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:52.950707 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff0b0dc-21ac-4261-b78c-05e40a50b9cb admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.008221 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.008892 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.009174 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.009450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] PATH_INFO: `/v3/projects/7e247930abd14072b8dfab4043d3e3f5` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.010583 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e247930abd14072b8dfab4043d3e3f5)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.012527 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.064119 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e247930abd14072b8dfab4043d3e3f5', 'request_id': 'req-4efb30dc-0e42-4821-a4a8-232198f91f13'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:53.070239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e247930abd14072b8dfab4043d3e3f5', 'request_id': 'req-4efb30dc-0e42-4821-a4a8-232198f91f13'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:53.075792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e247930abd14072b8dfab4043d3e3f5', 'request_id': 'req-4efb30dc-0e42-4821-a4a8-232198f91f13'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:53.089001 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] The token cache is being invalidate because project 7e247930abd14072b8dfab4043d3e3f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:46:53.089292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:53.089911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4efb30dc-0e42-4821-a4a8-232198f91f13 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:53.091439 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1397/2783] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:46:52 2022] DELETE /identity/v3/projects/7e247930abd14072b8dfab4043d3e3f5 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:53.097013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666bc7eb-290b-4d47-8d4d-f6353a1c0476 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.097378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666bc7eb-290b-4d47-8d4d-f6353a1c0476 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.097757 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666bc7eb-290b-4d47-8d4d-f6353a1c0476 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.154897 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-666bc7eb-290b-4d47-8d4d-f6353a1c0476 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:53.156132 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-666bc7eb-290b-4d47-8d4d-f6353a1c0476 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:53.198564 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:53.215569 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1387/2784] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:53.223715 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb257d-2ae0-4408-9ca3-e8380b262d40 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.224029 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb257d-2ae0-4408-9ca3-e8380b262d40 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.224308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb257d-2ae0-4408-9ca3-e8380b262d40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.244062 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-87bb257d-2ae0-4408-9ca3-e8380b262d40 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:53.245304 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-87bb257d-2ae0-4408-9ca3-e8380b262d40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:53.254070 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1398/2785] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:53.258278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666bc7eb-290b-4d47-8d4d-f6353a1c0476 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.269050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666bc7eb-290b-4d47-8d4d-f6353a1c0476 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.273742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fee824b-0477-425e-ab26-7c22ded87844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.274337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fee824b-0477-425e-ab26-7c22ded87844 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.274618 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fee824b-0477-425e-ab26-7c22ded87844 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.274895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fee824b-0477-425e-ab26-7c22ded87844 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.275541 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fee824b-0477-425e-ab26-7c22ded87844 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:46:53.275846 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fee824b-0477-425e-ab26-7c22ded87844 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.277501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fee824b-0477-425e-ab26-7c22ded87844 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.288943 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1388/2786] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:46:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:46:53.293208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bb257d-2ae0-4408-9ca3-e8380b262d40 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.302506 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bb257d-2ae0-4408-9ca3-e8380b262d40 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.307190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ee735d-5d58-4e32-a701-49a497c49562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.307785 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ee735d-5d58-4e32-a701-49a497c49562 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.308066 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ee735d-5d58-4e32-a701-49a497c49562 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.308353 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ee735d-5d58-4e32-a701-49a497c49562 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.309018 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ee735d-5d58-4e32-a701-49a497c49562 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.310920 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ee735d-5d58-4e32-a701-49a497c49562 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.347834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1399/2787] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:46:53 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 26 01:46:53.352013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fee824b-0477-425e-ab26-7c22ded87844 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.362721 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fee824b-0477-425e-ab26-7c22ded87844 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.417945 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542baec8-b2fe-4ace-b761-22fc8a7c8d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.418532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542baec8-b2fe-4ace-b761-22fc8a7c8d0f admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.418811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542baec8-b2fe-4ace-b761-22fc8a7c8d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.419085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542baec8-b2fe-4ace-b761-22fc8a7c8d0f admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.419720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542baec8-b2fe-4ace-b761-22fc8a7c8d0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.422219 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542baec8-b2fe-4ace-b761-22fc8a7c8d0f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.442552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1389/2788] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:46:53 2022] POST /identity/v3/users => generated 287 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:46:53.446861 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ee735d-5d58-4e32-a701-49a497c49562 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.457213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ee735d-5d58-4e32-a701-49a497c49562 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.461804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6cf9d0-a2b0-4eff-8559-bc4ba00fb35d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.462396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6cf9d0-a2b0-4eff-8559-bc4ba00fb35d admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.462675 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6cf9d0-a2b0-4eff-8559-bc4ba00fb35d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.462950 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6cf9d0-a2b0-4eff-8559-bc4ba00fb35d admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.463612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6cf9d0-a2b0-4eff-8559-bc4ba00fb35d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.465228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6cf9d0-a2b0-4eff-8559-bc4ba00fb35d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.475523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1400/2789] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:46:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:46:53.479476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542baec8-b2fe-4ace-b761-22fc8a7c8d0f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.490218 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542baec8-b2fe-4ace-b761-22fc8a7c8d0f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.495485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8f9dc5-ef58-4622-9e7d-da827369d641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.496069 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f9dc5-ef58-4622-9e7d-da827369d641 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.496361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f9dc5-ef58-4622-9e7d-da827369d641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.496655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8f9dc5-ef58-4622-9e7d-da827369d641 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.497291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8f9dc5-ef58-4622-9e7d-da827369d641 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.498870 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8f9dc5-ef58-4622-9e7d-da827369d641 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.509410 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1390/2790] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:46:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:46:53.513497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6cf9d0-a2b0-4eff-8559-bc4ba00fb35d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.524143 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6cf9d0-a2b0-4eff-8559-bc4ba00fb35d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.528836 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d86f97-3b70-4c70-9977-080d2dd14c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.529475 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d86f97-3b70-4c70-9977-080d2dd14c9b admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.529755 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d86f97-3b70-4c70-9977-080d2dd14c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.530030 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d86f97-3b70-4c70-9977-080d2dd14c9b admin admin] PATH_INFO: `/v3/projects/154e6052dc814350912e8bcc694b2659/users/a19d087dd0bb44b6a47735a4c1adc323/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.544913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d86f97-3b70-4c70-9977-080d2dd14c9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=154e6052dc814350912e8bcc694b2659, user_id=a19d087dd0bb44b6a47735a4c1adc323, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.547645 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d86f97-3b70-4c70-9977-080d2dd14c9b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.560039 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1401/2791] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:46:53 2022] PUT /identity/v3/projects/154e6052dc814350912e8bcc694b2659/users/a19d087dd0bb44b6a47735a4c1adc323/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:46:53.571466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60ef947-44c0-44b9-8582-6bcf0cd43361 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.571773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60ef947-44c0-44b9-8582-6bcf0cd43361 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.572066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60ef947-44c0-44b9-8582-6bcf0cd43361 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.600475 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c60ef947-44c0-44b9-8582-6bcf0cd43361 None None] MFA Rules not processed for user `a19d087dd0bb44b6a47735a4c1adc323`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:53.601616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c60ef947-44c0-44b9-8582-6bcf0cd43361 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:53.665900 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1391/2792] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:53.670625 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d86f97-3b70-4c70-9977-080d2dd14c9b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.690141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:53.734938 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d86f97-3b70-4c70-9977-080d2dd14c9b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.739547 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954156d6-4bbe-4f64-b019-383f1f67c25e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.740145 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954156d6-4bbe-4f64-b019-383f1f67c25e admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.740460 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954156d6-4bbe-4f64-b019-383f1f67c25e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.740732 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954156d6-4bbe-4f64-b019-383f1f67c25e admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.741364 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954156d6-4bbe-4f64-b019-383f1f67c25e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.743257 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954156d6-4bbe-4f64-b019-383f1f67c25e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.769252 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1402/2793] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:46:53 2022] POST /identity/v3/projects => generated 373 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:46:53.773274 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60ef947-44c0-44b9-8582-6bcf0cd43361 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.784820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60ef947-44c0-44b9-8582-6bcf0cd43361 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.826425 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:53.842693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00465c51-a73c-4b61-9b3a-3cfedf978ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.843291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00465c51-a73c-4b61-9b3a-3cfedf978ac9 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.844133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00465c51-a73c-4b61-9b3a-3cfedf978ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.844433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00465c51-a73c-4b61-9b3a-3cfedf978ac9 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.845058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00465c51-a73c-4b61-9b3a-3cfedf978ac9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.846957 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00465c51-a73c-4b61-9b3a-3cfedf978ac9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.867128 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1392/2794] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:46:53 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:46:53.871099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954156d6-4bbe-4f64-b019-383f1f67c25e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.881916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954156d6-4bbe-4f64-b019-383f1f67c25e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.886642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a90b71d-0356-4f96-b1d8-001bb8478145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.887240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a90b71d-0356-4f96-b1d8-001bb8478145 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.887523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a90b71d-0356-4f96-b1d8-001bb8478145 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.887794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a90b71d-0356-4f96-b1d8-001bb8478145 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.888470 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a90b71d-0356-4f96-b1d8-001bb8478145 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.890075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a90b71d-0356-4f96-b1d8-001bb8478145 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.901120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1403/2795] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:46:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:46:53.905225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00465c51-a73c-4b61-9b3a-3cfedf978ac9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.916860 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00465c51-a73c-4b61-9b3a-3cfedf978ac9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:53.922090 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18492595-a3e1-4fb2-90f6-44b77757c5de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:53.922894 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18492595-a3e1-4fb2-90f6-44b77757c5de admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:53.923171 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18492595-a3e1-4fb2-90f6-44b77757c5de admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:53.923444 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18492595-a3e1-4fb2-90f6-44b77757c5de admin admin] PATH_INFO: `/v3/projects/0b97933df1df4e4ab19d6ac20cf263b6/users/dc26f79514344900834e609e25d4cf7e/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:53.939611 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18492595-a3e1-4fb2-90f6-44b77757c5de admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b97933df1df4e4ab19d6ac20cf263b6, user_id=dc26f79514344900834e609e25d4cf7e, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:53.942290 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18492595-a3e1-4fb2-90f6-44b77757c5de admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:53.955325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1393/2796] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:46:53 2022] PUT /identity/v3/projects/0b97933df1df4e4ab19d6ac20cf263b6/users/dc26f79514344900834e609e25d4cf7e/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:53.959394 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a90b71d-0356-4f96-b1d8-001bb8478145 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:53.969396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a90b71d-0356-4f96-b1d8-001bb8478145 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:54.025133 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bdc06e-6226-4716-8856-746b2adaad14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:54.025729 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bdc06e-6226-4716-8856-746b2adaad14 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:54.026008 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bdc06e-6226-4716-8856-746b2adaad14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:54.026280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bdc06e-6226-4716-8856-746b2adaad14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:54.026912 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bdc06e-6226-4716-8856-746b2adaad14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:54.028550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bdc06e-6226-4716-8856-746b2adaad14 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:54.038312 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1404/2797] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:46:53 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:46:54.042731 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18492595-a3e1-4fb2-90f6-44b77757c5de admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:54.054207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18492595-a3e1-4fb2-90f6-44b77757c5de admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:54.058833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c73eb9-a578-4ae0-8a04-d0c1aed0c82f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:54.059509 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c73eb9-a578-4ae0-8a04-d0c1aed0c82f admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:54.059812 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c73eb9-a578-4ae0-8a04-d0c1aed0c82f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:54.060087 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c73eb9-a578-4ae0-8a04-d0c1aed0c82f admin admin] PATH_INFO: `/v3/projects/0b97933df1df4e4ab19d6ac20cf263b6/users/dc26f79514344900834e609e25d4cf7e/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:54.062016 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c73eb9-a578-4ae0-8a04-d0c1aed0c82f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b97933df1df4e4ab19d6ac20cf263b6, user_id=dc26f79514344900834e609e25d4cf7e, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:54.065291 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c73eb9-a578-4ae0-8a04-d0c1aed0c82f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:54.075352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1394/2798] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:46:54 2022] PUT /identity/v3/projects/0b97933df1df4e4ab19d6ac20cf263b6/users/dc26f79514344900834e609e25d4cf7e/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:54.086128 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb220a1e-cd0e-41b4-92f7-2b40808cec7c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:54.086414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb220a1e-cd0e-41b4-92f7-2b40808cec7c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:54.086685 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb220a1e-cd0e-41b4-92f7-2b40808cec7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:54.114043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-fb220a1e-cd0e-41b4-92f7-2b40808cec7c None None] MFA Rules not processed for user `dc26f79514344900834e609e25d4cf7e`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:54.115906 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fb220a1e-cd0e-41b4-92f7-2b40808cec7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:54.183702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1405/2799] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:54 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:54.195196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c73eb9-a578-4ae0-8a04-d0c1aed0c82f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:54.196933 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-81c73eb9-a578-4ae0-8a04-d0c1aed0c82f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:54.254918 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c73eb9-a578-4ae0-8a04-d0c1aed0c82f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:54.258524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0821d35f-a0e7-43bc-b39e-d2186b7a1f1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:54.259111 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0821d35f-a0e7-43bc-b39e-d2186b7a1f1b service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:54.259389 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0821d35f-a0e7-43bc-b39e-d2186b7a1f1b service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:54.259659 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0821d35f-a0e7-43bc-b39e-d2186b7a1f1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:54.268729 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0821d35f-a0e7-43bc-b39e-d2186b7a1f1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:54.270155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0821d35f-a0e7-43bc-b39e-d2186b7a1f1b service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:54.279481 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1395/2800] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:46:54 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:46:54.661407 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb220a1e-cd0e-41b4-92f7-2b40808cec7c None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:54.674900 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:54.725594 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb220a1e-cd0e-41b4-92f7-2b40808cec7c None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:54.730039 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:54.730794 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:54.731075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:54.731345 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] PATH_INFO: `/v3/users/a19d087dd0bb44b6a47735a4c1adc323` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:54.732490 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] RBAC: Authorizing `identity:delete_user(user_id=a19d087dd0bb44b6a47735a4c1adc323)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:54.734402 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:54.779833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a19d087dd0bb44b6a47735a4c1adc323'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:54.786596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a19d087dd0bb44b6a47735a4c1adc323'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:54.787210 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a19d087dd0bb44b6a47735a4c1adc323'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:54.787817 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a19d087dd0bb44b6a47735a4c1adc323'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:54.800280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a19d087dd0bb44b6a47735a4c1adc323'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:54.813227 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1406/2801] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:46:54 2022] DELETE /identity/v3/users/a19d087dd0bb44b6a47735a4c1adc323 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:54.817854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8a0248-bd8d-49a4-ad41-1d3c5a31180b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:54.818379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8a0248-bd8d-49a4-ad41-1d3c5a31180b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:54.818751 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8a0248-bd8d-49a4-ad41-1d3c5a31180b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:54.838205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fa8a0248-bd8d-49a4-ad41-1d3c5a31180b None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:54.839386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fa8a0248-bd8d-49a4-ad41-1d3c5a31180b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:54.895932 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1396/2802] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:54.905654 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:54.907676 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:54.964554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4595fd1c-193a-45c9-9d93-bb080a56d89d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:54.968183 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5509cb71-0e98-4ff4-a7f3-c72577e2ba42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:54.969334 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5509cb71-0e98-4ff4-a7f3-c72577e2ba42 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:54.969613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5509cb71-0e98-4ff4-a7f3-c72577e2ba42 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:54.969888 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5509cb71-0e98-4ff4-a7f3-c72577e2ba42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:54.978861 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5509cb71-0e98-4ff4-a7f3-c72577e2ba42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:54.981129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5509cb71-0e98-4ff4-a7f3-c72577e2ba42 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:54.991085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1407/2803] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:46:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:46:55.258489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8a0248-bd8d-49a4-ad41-1d3c5a31180b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:55.259979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fa8a0248-bd8d-49a4-ad41-1d3c5a31180b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:55.273229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8a0248-bd8d-49a4-ad41-1d3c5a31180b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:55.278010 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:55.278643 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:55.278926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:55.279196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] PATH_INFO: `/v3/projects/154e6052dc814350912e8bcc694b2659` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:55.280482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] RBAC: Authorizing `identity:delete_project(project_id=154e6052dc814350912e8bcc694b2659)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:55.283127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:55.320210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:55.332031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '154e6052dc814350912e8bcc694b2659', 'request_id': 'req-c5b3c114-7662-4892-ab47-97ce04fabe4c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.338255 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '154e6052dc814350912e8bcc694b2659', 'request_id': 'req-c5b3c114-7662-4892-ab47-97ce04fabe4c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.346654 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '154e6052dc814350912e8bcc694b2659', 'request_id': 'req-c5b3c114-7662-4892-ab47-97ce04fabe4c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.357096 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] The token cache is being invalidate because project 154e6052dc814350912e8bcc694b2659 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:46:55.357386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.358017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.359891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1397/2804] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:46:55 2022] DELETE /identity/v3/projects/154e6052dc814350912e8bcc694b2659 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:46:55.363652 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5509cb71-0e98-4ff4-a7f3-c72577e2ba42 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:55.373161 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5509cb71-0e98-4ff4-a7f3-c72577e2ba42 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:55.407501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:55.433465 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:55.434086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:55.434363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:55.434633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] PATH_INFO: `/v3/users/dc26f79514344900834e609e25d4cf7e` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:55.435730 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc26f79514344900834e609e25d4cf7e)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:55.438317 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:55.483067 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc26f79514344900834e609e25d4cf7e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.490613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc26f79514344900834e609e25d4cf7e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.491236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc26f79514344900834e609e25d4cf7e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.491843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc26f79514344900834e609e25d4cf7e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.504396 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-21ca9a2b-ddca-461b-b89f-18bae78c6681 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc26f79514344900834e609e25d4cf7e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.517153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1408/2805] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:46:55 2022] DELETE /identity/v3/users/dc26f79514344900834e609e25d4cf7e => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:55.785671 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:55.787184 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:55.800320 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b3c114-7662-4892-ab47-97ce04fabe4c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:55.860174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:55.860850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:55.861093 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:55.861369 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] PATH_INFO: `/v3/projects/0b97933df1df4e4ab19d6ac20cf263b6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:55.862496 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b97933df1df4e4ab19d6ac20cf263b6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:55.864431 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:55.910366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b97933df1df4e4ab19d6ac20cf263b6', 'request_id': 'req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.916560 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b97933df1df4e4ab19d6ac20cf263b6', 'request_id': 'req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.923748 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b97933df1df4e4ab19d6ac20cf263b6', 'request_id': 'req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.937046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] The token cache is being invalidate because project 0b97933df1df4e4ab19d6ac20cf263b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:46:55.937436 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.938407 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-099d8fa6-cbfc-4c52-bed6-16f42ad44ed3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:55.939879 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1398/2806] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:46:55 2022] DELETE /identity/v3/projects/0b97933df1df4e4ab19d6ac20cf263b6 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:55.944922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1912bd-6526-48ae-8e35-0a632f94475a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:55.945212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1912bd-6526-48ae-8e35-0a632f94475a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:55.945487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1912bd-6526-48ae-8e35-0a632f94475a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:55.965391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-7b1912bd-6526-48ae-8e35-0a632f94475a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:55.966596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-7b1912bd-6526-48ae-8e35-0a632f94475a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:56.024431 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1409/2807] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:56.033880 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01c1db-81b3-4be9-9c1e-f1a1a75be22a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.034102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01c1db-81b3-4be9-9c1e-f1a1a75be22a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.034376 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01c1db-81b3-4be9-9c1e-f1a1a75be22a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.055048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9f01c1db-81b3-4be9-9c1e-f1a1a75be22a None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:56.056275 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9f01c1db-81b3-4be9-9c1e-f1a1a75be22a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:56.065062 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1399/2808] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:46:56.069248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1912bd-6526-48ae-8e35-0a632f94475a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.079771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1912bd-6526-48ae-8e35-0a632f94475a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.084326 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e16e04-30e8-4d03-a070-fe088657c986 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.084934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e16e04-30e8-4d03-a070-fe088657c986 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.085212 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e16e04-30e8-4d03-a070-fe088657c986 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.085482 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e16e04-30e8-4d03-a070-fe088657c986 admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.086146 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e16e04-30e8-4d03-a070-fe088657c986 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:46:56.086445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e16e04-30e8-4d03-a070-fe088657c986 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.088040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e16e04-30e8-4d03-a070-fe088657c986 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.100264 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1410/2809] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:46:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:46:56.104357 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f01c1db-81b3-4be9-9c1e-f1a1a75be22a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.113613 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f01c1db-81b3-4be9-9c1e-f1a1a75be22a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.118170 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5049f9-6d94-4425-a574-d1a53ad43a22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.118749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5049f9-6d94-4425-a574-d1a53ad43a22 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.119024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5049f9-6d94-4425-a574-d1a53ad43a22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.119293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5049f9-6d94-4425-a574-d1a53ad43a22 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.119914 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5049f9-6d94-4425-a574-d1a53ad43a22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.121833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5049f9-6d94-4425-a574-d1a53ad43a22 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.149156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1400/2810] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:46:56 2022] POST /identity/v3/projects => generated 369 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:46:56.153211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e16e04-30e8-4d03-a070-fe088657c986 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.163013 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e16e04-30e8-4d03-a070-fe088657c986 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.204124 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:56.222079 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b737c559-f07b-49bc-a56e-f11929e3e2e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.222699 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b737c559-f07b-49bc-a56e-f11929e3e2e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.222962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b737c559-f07b-49bc-a56e-f11929e3e2e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.223236 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b737c559-f07b-49bc-a56e-f11929e3e2e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.223868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b737c559-f07b-49bc-a56e-f11929e3e2e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.225811 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b737c559-f07b-49bc-a56e-f11929e3e2e1 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.246213 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1411/2811] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:46:56 2022] POST /identity/v3/users => generated 285 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:46:56.251065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5049f9-6d94-4425-a574-d1a53ad43a22 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.261136 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5049f9-6d94-4425-a574-d1a53ad43a22 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.265850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58195193-77c4-482e-84b9-02c0130cafa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.266433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58195193-77c4-482e-84b9-02c0130cafa9 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.266715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58195193-77c4-482e-84b9-02c0130cafa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.266996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58195193-77c4-482e-84b9-02c0130cafa9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.267640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58195193-77c4-482e-84b9-02c0130cafa9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.269267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58195193-77c4-482e-84b9-02c0130cafa9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.279112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1401/2812] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:46:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:46:56.283771 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b737c559-f07b-49bc-a56e-f11929e3e2e1 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.293635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b737c559-f07b-49bc-a56e-f11929e3e2e1 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.298833 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2db954-919a-4f67-a615-0e29bc051303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.299415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2db954-919a-4f67-a615-0e29bc051303 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.299690 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2db954-919a-4f67-a615-0e29bc051303 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.299958 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2db954-919a-4f67-a615-0e29bc051303 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.300624 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2db954-919a-4f67-a615-0e29bc051303 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.302205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2db954-919a-4f67-a615-0e29bc051303 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.313250 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1412/2813] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:46:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:46:56.317292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58195193-77c4-482e-84b9-02c0130cafa9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.327023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58195193-77c4-482e-84b9-02c0130cafa9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.331715 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f75c771-3714-4ac8-b51a-e6d544dd1113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.332378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f75c771-3714-4ac8-b51a-e6d544dd1113 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.332655 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f75c771-3714-4ac8-b51a-e6d544dd1113 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.332928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f75c771-3714-4ac8-b51a-e6d544dd1113 admin admin] PATH_INFO: `/v3/projects/20d618b05a90402aaf78c024c8f7c226/users/b9dc92f3dd2c448eb837fd150994faa8/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.344382 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:56.348974 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f75c771-3714-4ac8-b51a-e6d544dd1113 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20d618b05a90402aaf78c024c8f7c226, user_id=b9dc92f3dd2c448eb837fd150994faa8, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.352113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f75c771-3714-4ac8-b51a-e6d544dd1113 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.366366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1402/2814] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:46:56 2022] PUT /identity/v3/projects/20d618b05a90402aaf78c024c8f7c226/users/b9dc92f3dd2c448eb837fd150994faa8/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:56.377087 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8bd17a-8953-42b8-a875-9039ec24e07c None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.377376 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8bd17a-8953-42b8-a875-9039ec24e07c None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.377647 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8bd17a-8953-42b8-a875-9039ec24e07c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.404609 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-5b8bd17a-8953-42b8-a875-9039ec24e07c None None] MFA Rules not processed for user `b9dc92f3dd2c448eb837fd150994faa8`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:56.405835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5b8bd17a-8953-42b8-a875-9039ec24e07c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:56.469775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1413/2815] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:56 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 26 01:46:56.474379 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f75c771-3714-4ac8-b51a-e6d544dd1113 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.536539 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f75c771-3714-4ac8-b51a-e6d544dd1113 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.541118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb3d418-cb9a-4c85-8804-292d2f1ca685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.541699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb3d418-cb9a-4c85-8804-292d2f1ca685 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.541976 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb3d418-cb9a-4c85-8804-292d2f1ca685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.542250 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb3d418-cb9a-4c85-8804-292d2f1ca685 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.542872 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb3d418-cb9a-4c85-8804-292d2f1ca685 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.544849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb3d418-cb9a-4c85-8804-292d2f1ca685 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.570902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1403/2816] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:46:56 2022] POST /identity/v3/projects => generated 367 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:46:56.574990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8bd17a-8953-42b8-a875-9039ec24e07c None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.586125 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8bd17a-8953-42b8-a875-9039ec24e07c None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.642960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e03b11-d9e8-42ee-9293-e6a431b7cbd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.643567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e03b11-d9e8-42ee-9293-e6a431b7cbd3 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.643850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e03b11-d9e8-42ee-9293-e6a431b7cbd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.644127 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e03b11-d9e8-42ee-9293-e6a431b7cbd3 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.644791 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e03b11-d9e8-42ee-9293-e6a431b7cbd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.646705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e03b11-d9e8-42ee-9293-e6a431b7cbd3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.669691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1414/2817] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:46:56 2022] POST /identity/v3/users => generated 290 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:46:56.672569 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb3d418-cb9a-4c85-8804-292d2f1ca685 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.682719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb3d418-cb9a-4c85-8804-292d2f1ca685 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.687348 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c156adf-b841-428f-a8ba-9a4968458731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.688305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c156adf-b841-428f-a8ba-9a4968458731 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.688617 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c156adf-b841-428f-a8ba-9a4968458731 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.688888 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c156adf-b841-428f-a8ba-9a4968458731 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.689528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c156adf-b841-428f-a8ba-9a4968458731 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.691112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c156adf-b841-428f-a8ba-9a4968458731 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.702310 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1404/2818] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:46:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:46:56.707040 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e03b11-d9e8-42ee-9293-e6a431b7cbd3 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.716972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e03b11-d9e8-42ee-9293-e6a431b7cbd3 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.722255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57b744a-6c12-4355-8daf-00b19ebb1083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.722902 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57b744a-6c12-4355-8daf-00b19ebb1083 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.723182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57b744a-6c12-4355-8daf-00b19ebb1083 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.723455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57b744a-6c12-4355-8daf-00b19ebb1083 admin admin] PATH_INFO: `/v3/projects/38d2b795bebe4164be8f4d8c8a3df7b8/users/94955dd91b434daca9655cfcb2bd30b7/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.738176 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57b744a-6c12-4355-8daf-00b19ebb1083 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38d2b795bebe4164be8f4d8c8a3df7b8, user_id=94955dd91b434daca9655cfcb2bd30b7, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.740926 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57b744a-6c12-4355-8daf-00b19ebb1083 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.753242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1415/2819] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:46:56 2022] PUT /identity/v3/projects/38d2b795bebe4164be8f4d8c8a3df7b8/users/94955dd91b434daca9655cfcb2bd30b7/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:56.757095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c156adf-b841-428f-a8ba-9a4968458731 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.767583 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c156adf-b841-428f-a8ba-9a4968458731 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.823323 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4efd77-aed4-4ba3-9721-acc8120b15fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.823862 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4efd77-aed4-4ba3-9721-acc8120b15fe admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.824202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4efd77-aed4-4ba3-9721-acc8120b15fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.824503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4efd77-aed4-4ba3-9721-acc8120b15fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.825251 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4efd77-aed4-4ba3-9721-acc8120b15fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.826840 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4efd77-aed4-4ba3-9721-acc8120b15fe admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.837288 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1405/2820] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:46:56 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:46:56.841434 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57b744a-6c12-4355-8daf-00b19ebb1083 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:56.852881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57b744a-6c12-4355-8daf-00b19ebb1083 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:56.857466 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178d3fd2-1684-4088-bb32-685533ddd58e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:56.858115 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178d3fd2-1684-4088-bb32-685533ddd58e admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.858392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178d3fd2-1684-4088-bb32-685533ddd58e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.858662 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178d3fd2-1684-4088-bb32-685533ddd58e admin admin] PATH_INFO: `/v3/projects/38d2b795bebe4164be8f4d8c8a3df7b8/users/94955dd91b434daca9655cfcb2bd30b7/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.860658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178d3fd2-1684-4088-bb32-685533ddd58e admin admin] RBAC: Authorizing `identity:create_grant(project_id=38d2b795bebe4164be8f4d8c8a3df7b8, user_id=94955dd91b434daca9655cfcb2bd30b7, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:56.863279 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178d3fd2-1684-4088-bb32-685533ddd58e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:56.873365 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1416/2821] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:46:56 2022] PUT /identity/v3/projects/38d2b795bebe4164be8f4d8c8a3df7b8/users/94955dd91b434daca9655cfcb2bd30b7/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:56.883871 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9329c84f-0e7d-4193-82fc-0abb9215647c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:56.884386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9329c84f-0e7d-4193-82fc-0abb9215647c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:56.884502 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9329c84f-0e7d-4193-82fc-0abb9215647c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:56.912892 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9329c84f-0e7d-4193-82fc-0abb9215647c None None] MFA Rules not processed for user `94955dd91b434daca9655cfcb2bd30b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:56.914344 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9329c84f-0e7d-4193-82fc-0abb9215647c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:57.009424 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:57.031816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1406/2822] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:56 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:57.042981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178d3fd2-1684-4088-bb32-685533ddd58e admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:57.044722 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-178d3fd2-1684-4088-bb32-685533ddd58e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:57.051274 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:57.105834 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178d3fd2-1684-4088-bb32-685533ddd58e admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:57.109863 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc562722-8c6d-4d71-8fac-93b6251ddf31 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:57.110450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc562722-8c6d-4d71-8fac-93b6251ddf31 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:57.110723 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc562722-8c6d-4d71-8fac-93b6251ddf31 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:57.110987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc562722-8c6d-4d71-8fac-93b6251ddf31 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:57.119835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc562722-8c6d-4d71-8fac-93b6251ddf31 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:57.121262 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc562722-8c6d-4d71-8fac-93b6251ddf31 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:57.131109 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1417/2823] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:46:57 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:46:58.467743 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9329c84f-0e7d-4193-82fc-0abb9215647c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:58.529463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9329c84f-0e7d-4193-82fc-0abb9215647c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:58.534220 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:58.534843 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:58.535124 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:58.535404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] PATH_INFO: `/v3/users/b9dc92f3dd2c448eb837fd150994faa8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:58.536558 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9dc92f3dd2c448eb837fd150994faa8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:58.538479 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:58.583920 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9dc92f3dd2c448eb837fd150994faa8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:58.591361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9dc92f3dd2c448eb837fd150994faa8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:58.591987 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9dc92f3dd2c448eb837fd150994faa8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:58.592660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9dc92f3dd2c448eb837fd150994faa8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:58.599995 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9dc92f3dd2c448eb837fd150994faa8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:58.618095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1407/2824] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:46:58 2022] DELETE /identity/v3/users/b9dc92f3dd2c448eb837fd150994faa8 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:58.622352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab1703d-ab31-486d-abc4-7cdf4210b265 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:58.622807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab1703d-ab31-486d-abc4-7cdf4210b265 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:58.622910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab1703d-ab31-486d-abc4-7cdf4210b265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:58.642305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-fab1703d-ab31-486d-abc4-7cdf4210b265 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:58.643445 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fab1703d-ab31-486d-abc4-7cdf4210b265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:58.701141 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1418/2825] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:58.710895 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:58.712374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:58.768314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb466d37-2914-478d-8028-9d0b5f4a7e00 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:58.771921 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b587c68c-22b2-4366-ba21-5b503d5ad4c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:58.772544 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b587c68c-22b2-4366-ba21-5b503d5ad4c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:58.772821 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b587c68c-22b2-4366-ba21-5b503d5ad4c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:58.773088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b587c68c-22b2-4366-ba21-5b503d5ad4c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:58.781850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b587c68c-22b2-4366-ba21-5b503d5ad4c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:58.783272 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b587c68c-22b2-4366-ba21-5b503d5ad4c8 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:58.792691 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1408/2826] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:46:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:46:59.057308 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab1703d-ab31-486d-abc4-7cdf4210b265 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:59.058744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-fab1703d-ab31-486d-abc4-7cdf4210b265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:59.071194 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab1703d-ab31-486d-abc4-7cdf4210b265 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:59.075988 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:59.076655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:59.076936 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:59.077209 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] PATH_INFO: `/v3/projects/20d618b05a90402aaf78c024c8f7c226` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:59.078352 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] RBAC: Authorizing `identity:delete_project(project_id=20d618b05a90402aaf78c024c8f7c226)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:59.080284 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:59.128179 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20d618b05a90402aaf78c024c8f7c226', 'request_id': 'req-4dcb0039-3804-446a-b784-dec8d4442641'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.134159 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20d618b05a90402aaf78c024c8f7c226', 'request_id': 'req-4dcb0039-3804-446a-b784-dec8d4442641'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.139600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20d618b05a90402aaf78c024c8f7c226', 'request_id': 'req-4dcb0039-3804-446a-b784-dec8d4442641'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.151789 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] The token cache is being invalidate because project 20d618b05a90402aaf78c024c8f7c226 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:46:59.152074 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.152701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.154196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1419/2827] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:46:59 2022] DELETE /identity/v3/projects/20d618b05a90402aaf78c024c8f7c226 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:59.158947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b587c68c-22b2-4366-ba21-5b503d5ad4c8 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:59.168518 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b587c68c-22b2-4366-ba21-5b503d5ad4c8 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:59.172823 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:59.227386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:59.228008 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:59.228292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:59.228598 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] PATH_INFO: `/v3/users/94955dd91b434daca9655cfcb2bd30b7` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:59.229714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] RBAC: Authorizing `identity:delete_user(user_id=94955dd91b434daca9655cfcb2bd30b7)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:59.231649 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:59.280082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94955dd91b434daca9655cfcb2bd30b7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.288940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94955dd91b434daca9655cfcb2bd30b7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.289660 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94955dd91b434daca9655cfcb2bd30b7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.290259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94955dd91b434daca9655cfcb2bd30b7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.297998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4414eff6-36b3-45aa-84e7-91588a6a1caa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94955dd91b434daca9655cfcb2bd30b7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.315877 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1409/2828] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:46:59 2022] DELETE /identity/v3/users/94955dd91b434daca9655cfcb2bd30b7 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:59.592090 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:59.594650 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:59.608217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcb0039-3804-446a-b784-dec8d4442641 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:59.636612 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:46:59.669400 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:59.670026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:59.670301 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:59.670571 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] PATH_INFO: `/v3/projects/38d2b795bebe4164be8f4d8c8a3df7b8` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:59.671670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=38d2b795bebe4164be8f4d8c8a3df7b8)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:59.673987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:59.721493 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38d2b795bebe4164be8f4d8c8a3df7b8', 'request_id': 'req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.727539 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38d2b795bebe4164be8f4d8c8a3df7b8', 'request_id': 'req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.733073 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38d2b795bebe4164be8f4d8c8a3df7b8', 'request_id': 'req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.745098 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] The token cache is being invalidate because project 38d2b795bebe4164be8f4d8c8a3df7b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:46:59.745970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.746606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-299a1ec1-e439-474e-b4c4-9bb21bbdfd7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:46:59.748077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1420/2829] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:46:59 2022] DELETE /identity/v3/projects/38d2b795bebe4164be8f4d8c8a3df7b8 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:46:59.753594 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d763bef-aa98-40a5-8145-339a7bda5435 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:59.754094 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d763bef-aa98-40a5-8145-339a7bda5435 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:59.754202 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d763bef-aa98-40a5-8145-339a7bda5435 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:59.774050 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2d763bef-aa98-40a5-8145-339a7bda5435 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:59.775168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2d763bef-aa98-40a5-8145-339a7bda5435 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:59.834392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1410/2830] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:59.895235 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ff0452-75a2-45fc-bb1e-90cceffc51e6 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:59.895639 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ff0452-75a2-45fc-bb1e-90cceffc51e6 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:59.895793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ff0452-75a2-45fc-bb1e-90cceffc51e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:59.916338 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-61ff0452-75a2-45fc-bb1e-90cceffc51e6 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:46:59.917622 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-61ff0452-75a2-45fc-bb1e-90cceffc51e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:46:59.925498 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1421/2831] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:46:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:46:59.929670 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d763bef-aa98-40a5-8145-339a7bda5435 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:59.939999 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d763bef-aa98-40a5-8145-339a7bda5435 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:59.944765 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca04c67d-af85-40d3-bd7b-fe96c0386b6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:59.945368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca04c67d-af85-40d3-bd7b-fe96c0386b6e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:59.945641 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca04c67d-af85-40d3-bd7b-fe96c0386b6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:59.945913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca04c67d-af85-40d3-bd7b-fe96c0386b6e admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:59.946561 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca04c67d-af85-40d3-bd7b-fe96c0386b6e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:46:59.946863 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca04c67d-af85-40d3-bd7b-fe96c0386b6e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:59.948503 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca04c67d-af85-40d3-bd7b-fe96c0386b6e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:46:59.960885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1411/2832] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:46:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:46:59.965216 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ff0452-75a2-45fc-bb1e-90cceffc51e6 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:46:59.974449 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ff0452-75a2-45fc-bb1e-90cceffc51e6 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:46:59.979058 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0804cfe3-d801-40dd-8641-c543b7e7a711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:46:59.979642 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0804cfe3-d801-40dd-8641-c543b7e7a711 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:46:59.979916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0804cfe3-d801-40dd-8641-c543b7e7a711 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:46:59.980186 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0804cfe3-d801-40dd-8641-c543b7e7a711 admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:46:59.980830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0804cfe3-d801-40dd-8641-c543b7e7a711 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:46:59.982713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0804cfe3-d801-40dd-8641-c543b7e7a711 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.009280 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1422/2833] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:46:59 2022] POST /identity/v3/projects => generated 375 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:00.013353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca04c67d-af85-40d3-bd7b-fe96c0386b6e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.024153 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca04c67d-af85-40d3-bd7b-fe96c0386b6e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.033300 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:00.083181 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd963d07-ae3b-45a3-a662-598be84c0417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.083782 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd963d07-ae3b-45a3-a662-598be84c0417 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.084066 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd963d07-ae3b-45a3-a662-598be84c0417 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.084338 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd963d07-ae3b-45a3-a662-598be84c0417 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.084993 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd963d07-ae3b-45a3-a662-598be84c0417 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.086946 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd963d07-ae3b-45a3-a662-598be84c0417 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.107402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1412/2834] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:47:00 2022] POST /identity/v3/users => generated 288 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:00.112154 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0804cfe3-d801-40dd-8641-c543b7e7a711 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.122987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0804cfe3-d801-40dd-8641-c543b7e7a711 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.128419 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061733ac-35c7-4615-898b-c957ca4aaab8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.129014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061733ac-35c7-4615-898b-c957ca4aaab8 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.129295 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061733ac-35c7-4615-898b-c957ca4aaab8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.129570 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061733ac-35c7-4615-898b-c957ca4aaab8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.130211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061733ac-35c7-4615-898b-c957ca4aaab8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.131842 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061733ac-35c7-4615-898b-c957ca4aaab8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.141706 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1423/2835] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:00.145610 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd963d07-ae3b-45a3-a662-598be84c0417 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.155458 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd963d07-ae3b-45a3-a662-598be84c0417 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.160031 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4570cd52-e2b1-41d8-98b7-3dd044863da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.160653 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4570cd52-e2b1-41d8-98b7-3dd044863da2 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.160936 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4570cd52-e2b1-41d8-98b7-3dd044863da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.161210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4570cd52-e2b1-41d8-98b7-3dd044863da2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.161854 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4570cd52-e2b1-41d8-98b7-3dd044863da2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.163454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4570cd52-e2b1-41d8-98b7-3dd044863da2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.175595 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1413/2836] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:00.179504 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061733ac-35c7-4615-898b-c957ca4aaab8 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.189860 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061733ac-35c7-4615-898b-c957ca4aaab8 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.194710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c2f3b5-101f-4b91-a9a2-992baf824206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.195404 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c2f3b5-101f-4b91-a9a2-992baf824206 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.195686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c2f3b5-101f-4b91-a9a2-992baf824206 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.195963 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c2f3b5-101f-4b91-a9a2-992baf824206 admin admin] PATH_INFO: `/v3/projects/5d7bfc99d8f34bbd9f9f5763e66ae754/users/ef31094291144612818036ff2fa35712/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.210341 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c2f3b5-101f-4b91-a9a2-992baf824206 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d7bfc99d8f34bbd9f9f5763e66ae754, user_id=ef31094291144612818036ff2fa35712, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.213041 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c2f3b5-101f-4b91-a9a2-992baf824206 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.225421 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1424/2837] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:47:00 2022] PUT /identity/v3/projects/5d7bfc99d8f34bbd9f9f5763e66ae754/users/ef31094291144612818036ff2fa35712/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:00.236133 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586bfdda-9b3c-46a2-97a0-40d5068aaa7d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.236525 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586bfdda-9b3c-46a2-97a0-40d5068aaa7d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.236825 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586bfdda-9b3c-46a2-97a0-40d5068aaa7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.264293 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-586bfdda-9b3c-46a2-97a0-40d5068aaa7d None None] MFA Rules not processed for user `ef31094291144612818036ff2fa35712`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:00.265568 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-586bfdda-9b3c-46a2-97a0-40d5068aaa7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:00.332188 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1414/2838] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:00 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:00.336726 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c2f3b5-101f-4b91-a9a2-992baf824206 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.392298 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:00.398679 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c2f3b5-101f-4b91-a9a2-992baf824206 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.404259 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70ef7bc-5d6c-4182-8660-b5d1eea5af8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.404879 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70ef7bc-5d6c-4182-8660-b5d1eea5af8f admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.405153 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70ef7bc-5d6c-4182-8660-b5d1eea5af8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.405422 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70ef7bc-5d6c-4182-8660-b5d1eea5af8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.406085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70ef7bc-5d6c-4182-8660-b5d1eea5af8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.407979 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70ef7bc-5d6c-4182-8660-b5d1eea5af8f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.434009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1425/2839] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:47:00 2022] POST /identity/v3/projects => generated 377 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:00.439103 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586bfdda-9b3c-46a2-97a0-40d5068aaa7d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.449732 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586bfdda-9b3c-46a2-97a0-40d5068aaa7d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.506002 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dcf9c2-d892-4315-a561-4de59fdbf6e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.506603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dcf9c2-d892-4315-a561-4de59fdbf6e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.506881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dcf9c2-d892-4315-a561-4de59fdbf6e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.507155 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dcf9c2-d892-4315-a561-4de59fdbf6e8 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.507786 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1dcf9c2-d892-4315-a561-4de59fdbf6e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.509795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1dcf9c2-d892-4315-a561-4de59fdbf6e8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.531636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1415/2840] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:47:00 2022] POST /identity/v3/users => generated 295 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:00.535962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70ef7bc-5d6c-4182-8660-b5d1eea5af8f admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.546288 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70ef7bc-5d6c-4182-8660-b5d1eea5af8f admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.551616 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac838089-ea62-4017-96fc-534ae638c6e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.552211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac838089-ea62-4017-96fc-534ae638c6e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.552531 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac838089-ea62-4017-96fc-534ae638c6e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.552807 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac838089-ea62-4017-96fc-534ae638c6e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.553469 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac838089-ea62-4017-96fc-534ae638c6e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.555116 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac838089-ea62-4017-96fc-534ae638c6e2 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.565856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1426/2841] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:00.569953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dcf9c2-d892-4315-a561-4de59fdbf6e8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.580150 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dcf9c2-d892-4315-a561-4de59fdbf6e8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.584717 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedb21e0-47cf-4670-9981-f7e4ccbbfb26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.585355 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedb21e0-47cf-4670-9981-f7e4ccbbfb26 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.585631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedb21e0-47cf-4670-9981-f7e4ccbbfb26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.585902 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedb21e0-47cf-4670-9981-f7e4ccbbfb26 admin admin] PATH_INFO: `/v3/projects/94c5c93dce694e2f845be44269a43945/users/0876d20044f7465fb91039c2216d1bc7/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.595033 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:00.602730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedb21e0-47cf-4670-9981-f7e4ccbbfb26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94c5c93dce694e2f845be44269a43945, user_id=0876d20044f7465fb91039c2216d1bc7, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.605454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedb21e0-47cf-4670-9981-f7e4ccbbfb26 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.618829 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1416/2842] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:47:00 2022] PUT /identity/v3/projects/94c5c93dce694e2f845be44269a43945/users/0876d20044f7465fb91039c2216d1bc7/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:00.622980 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac838089-ea62-4017-96fc-534ae638c6e2 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.633696 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac838089-ea62-4017-96fc-534ae638c6e2 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.690324 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5349247c-5267-4ebd-846b-8293148786a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.690925 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5349247c-5267-4ebd-846b-8293148786a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.691205 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5349247c-5267-4ebd-846b-8293148786a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.691477 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5349247c-5267-4ebd-846b-8293148786a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.692228 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5349247c-5267-4ebd-846b-8293148786a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.693998 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5349247c-5267-4ebd-846b-8293148786a8 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.703808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1427/2843] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:00 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:00.707979 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedb21e0-47cf-4670-9981-f7e4ccbbfb26 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.718156 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedb21e0-47cf-4670-9981-f7e4ccbbfb26 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.723205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c257886-0608-4f13-b2f7-9c36d1b07b32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.723839 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c257886-0608-4f13-b2f7-9c36d1b07b32 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.724119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c257886-0608-4f13-b2f7-9c36d1b07b32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.724419 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c257886-0608-4f13-b2f7-9c36d1b07b32 admin admin] PATH_INFO: `/v3/projects/94c5c93dce694e2f845be44269a43945/users/0876d20044f7465fb91039c2216d1bc7/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.726386 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c257886-0608-4f13-b2f7-9c36d1b07b32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94c5c93dce694e2f845be44269a43945, user_id=0876d20044f7465fb91039c2216d1bc7, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.729079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c257886-0608-4f13-b2f7-9c36d1b07b32 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.739602 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1417/2844] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:47:00 2022] PUT /identity/v3/projects/94c5c93dce694e2f845be44269a43945/users/0876d20044f7465fb91039c2216d1bc7/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:00.751933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c88962a-378e-49ce-8011-049843e9b68a None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.752200 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c88962a-378e-49ce-8011-049843e9b68a None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.752630 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c88962a-378e-49ce-8011-049843e9b68a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.780265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-8c88962a-378e-49ce-8011-049843e9b68a None None] MFA Rules not processed for user `0876d20044f7465fb91039c2216d1bc7`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:00.782060 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8c88962a-378e-49ce-8011-049843e9b68a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:00.849974 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1428/2845] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:00 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:00.860848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c257886-0608-4f13-b2f7-9c36d1b07b32 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:00.862594 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0c257886-0608-4f13-b2f7-9c36d1b07b32 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:00.920185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c257886-0608-4f13-b2f7-9c36d1b07b32 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:00.924040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04307e4a-3a70-41c3-a36d-c578ca12d5f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:00.924664 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04307e4a-3a70-41c3-a36d-c578ca12d5f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:00.924941 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04307e4a-3a70-41c3-a36d-c578ca12d5f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:00.925209 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04307e4a-3a70-41c3-a36d-c578ca12d5f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:00.980837 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04307e4a-3a70-41c3-a36d-c578ca12d5f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:00.982270 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04307e4a-3a70-41c3-a36d-c578ca12d5f9 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:00.990994 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1418/2846] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:00 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:47:06.644204 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c88962a-378e-49ce-8011-049843e9b68a None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:06.645670 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8c88962a-378e-49ce-8011-049843e9b68a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:06.657962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c88962a-378e-49ce-8011-049843e9b68a None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:06.661501 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7f6d0a-ca51-4c18-b730-d4dcfd1f4c21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:06.662093 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f6d0a-ca51-4c18-b730-d4dcfd1f4c21 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:06.662372 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f6d0a-ca51-4c18-b730-d4dcfd1f4c21 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:06.662647 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f6d0a-ca51-4c18-b730-d4dcfd1f4c21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:06.671613 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7f6d0a-ca51-4c18-b730-d4dcfd1f4c21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:06.673120 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7f6d0a-ca51-4c18-b730-d4dcfd1f4c21 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:06.682550 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1429/2847] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:06 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:47:10.986996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=58,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:16.678253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=82,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:18.882210 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04307e4a-3a70-41c3-a36d-c578ca12d5f9 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:18.883778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-04307e4a-3a70-41c3-a36d-c578ca12d5f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:18.940545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04307e4a-3a70-41c3-a36d-c578ca12d5f9 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:18.944102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f575de9-8745-4393-9834-2ed476f50bb0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:18.944740 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f575de9-8745-4393-9834-2ed476f50bb0 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:18.945023 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f575de9-8745-4393-9834-2ed476f50bb0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:18.945299 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f575de9-8745-4393-9834-2ed476f50bb0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:18.947107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6f575de9-8745-4393-9834-2ed476f50bb0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:18.956581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f575de9-8745-4393-9834-2ed476f50bb0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:18.958007 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f575de9-8745-4393-9834-2ed476f50bb0 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:18.966040 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1419/2848] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:47:27.421178 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7f6d0a-ca51-4c18-b730-d4dcfd1f4c21 service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:27.482641 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7f6d0a-ca51-4c18-b730-d4dcfd1f4c21 service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:27.487356 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:27.487970 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:27.488244 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:27.488549 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] PATH_INFO: `/v3/users/ef31094291144612818036ff2fa35712` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:27.489646 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef31094291144612818036ff2fa35712)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:27.491549 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:27.537537 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef31094291144612818036ff2fa35712'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:27.544668 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef31094291144612818036ff2fa35712'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:27.545292 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef31094291144612818036ff2fa35712'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:27.545903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef31094291144612818036ff2fa35712'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:27.557916 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef31094291144612818036ff2fa35712'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:27.570764 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1430/2849] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:47:27 2022] DELETE /identity/v3/users/ef31094291144612818036ff2fa35712 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:27.575180 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bbea4a-e3f1-43ce-9511-c31323502c38 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:27.575675 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bbea4a-e3f1-43ce-9511-c31323502c38 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:27.575791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bbea4a-e3f1-43ce-9511-c31323502c38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:27.594778 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e2bbea4a-e3f1-43ce-9511-c31323502c38 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:27.596035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e2bbea4a-e3f1-43ce-9511-c31323502c38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:27.653885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1420/2850] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:27.663536 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:27.664987 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:27.722247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b472ba-81a1-49bc-bb1a-7f1305ad1160 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:27.725924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2fec47-5fc2-4498-951b-13c6d64e3612 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:27.726521 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2fec47-5fc2-4498-951b-13c6d64e3612 service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:27.726802 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2fec47-5fc2-4498-951b-13c6d64e3612 service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:27.727080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2fec47-5fc2-4498-951b-13c6d64e3612 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:27.736793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2fec47-5fc2-4498-951b-13c6d64e3612 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:27.738215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2fec47-5fc2-4498-951b-13c6d64e3612 service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:27.747568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1431/2851] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:47:28.008881 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bbea4a-e3f1-43ce-9511-c31323502c38 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:28.010366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e2bbea4a-e3f1-43ce-9511-c31323502c38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:28.025912 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bbea4a-e3f1-43ce-9511-c31323502c38 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:28.030631 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:28.031267 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:28.031545 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:28.031817 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] PATH_INFO: `/v3/projects/5d7bfc99d8f34bbd9f9f5763e66ae754` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:28.033816 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d7bfc99d8f34bbd9f9f5763e66ae754)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:28.039791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:28.088462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d7bfc99d8f34bbd9f9f5763e66ae754', 'request_id': 'req-04634c09-1c54-47b1-a41c-5965fecf72d1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.094612 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d7bfc99d8f34bbd9f9f5763e66ae754', 'request_id': 'req-04634c09-1c54-47b1-a41c-5965fecf72d1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.103233 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d7bfc99d8f34bbd9f9f5763e66ae754', 'request_id': 'req-04634c09-1c54-47b1-a41c-5965fecf72d1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.113971 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] The token cache is being invalidate because project 5d7bfc99d8f34bbd9f9f5763e66ae754 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:47:28.114259 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.114878 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.116352 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1421/2852] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:47:28 2022] DELETE /identity/v3/projects/5d7bfc99d8f34bbd9f9f5763e66ae754 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:28.120325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2fec47-5fc2-4498-951b-13c6d64e3612 service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:28.129981 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2fec47-5fc2-4498-951b-13c6d64e3612 service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:28.148516 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:28.189222 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:28.189846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:28.190126 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:28.190391 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] PATH_INFO: `/v3/users/0876d20044f7465fb91039c2216d1bc7` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:28.191502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0876d20044f7465fb91039c2216d1bc7)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:28.193461 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:28.238515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0876d20044f7465fb91039c2216d1bc7'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.246994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0876d20044f7465fb91039c2216d1bc7'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.247643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0876d20044f7465fb91039c2216d1bc7'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.248247 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0876d20044f7465fb91039c2216d1bc7'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.260985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-39d02484-bdda-4943-9510-c36ec51a54b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0876d20044f7465fb91039c2216d1bc7'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.273727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1432/2853] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:47:28 2022] DELETE /identity/v3/users/0876d20044f7465fb91039c2216d1bc7 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:28.542417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:28.544131 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:28.558546 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04634c09-1c54-47b1-a41c-5965fecf72d1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:28.584405 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:28.617851 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:28.618422 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:28.618699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:28.618966 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] PATH_INFO: `/v3/projects/94c5c93dce694e2f845be44269a43945` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:28.620078 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] RBAC: Authorizing `identity:delete_project(project_id=94c5c93dce694e2f845be44269a43945)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:28.622060 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:28.667287 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94c5c93dce694e2f845be44269a43945', 'request_id': 'req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.673896 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94c5c93dce694e2f845be44269a43945', 'request_id': 'req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.681353 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94c5c93dce694e2f845be44269a43945', 'request_id': 'req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.692459 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] The token cache is being invalidate because project 94c5c93dce694e2f845be44269a43945 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:47:28.692747 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.693400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f7c7ba67-85e3-4e4e-99eb-dc3937c43514 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:28.694869 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1422/2854] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:47:28 2022] DELETE /identity/v3/projects/94c5c93dce694e2f845be44269a43945 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:28.699850 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d00aa99-b06d-4d94-b5e6-d1752170d0e4 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:28.700360 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d00aa99-b06d-4d94-b5e6-d1752170d0e4 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:28.700472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d00aa99-b06d-4d94-b5e6-d1752170d0e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:28.719806 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-9d00aa99-b06d-4d94-b5e6-d1752170d0e4 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:28.721075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9d00aa99-b06d-4d94-b5e6-d1752170d0e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:28.778911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1433/2855] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:28.786917 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5c9c49-9489-4035-9306-dff36bca417e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:28.787229 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5c9c49-9489-4035-9306-dff36bca417e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:28.787507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5c9c49-9489-4035-9306-dff36bca417e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:28.807465 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-db5c9c49-9489-4035-9306-dff36bca417e None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:28.808705 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-db5c9c49-9489-4035-9306-dff36bca417e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:28.817784 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1423/2856] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:28.822555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d00aa99-b06d-4d94-b5e6-d1752170d0e4 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:28.833426 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d00aa99-b06d-4d94-b5e6-d1752170d0e4 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:28.838094 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b58d23-6e2e-4ab1-8020-0437ef43894f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:28.838691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b58d23-6e2e-4ab1-8020-0437ef43894f admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:28.838973 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b58d23-6e2e-4ab1-8020-0437ef43894f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:28.839248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b58d23-6e2e-4ab1-8020-0437ef43894f admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:28.839895 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b58d23-6e2e-4ab1-8020-0437ef43894f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:47:28.840202 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b58d23-6e2e-4ab1-8020-0437ef43894f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:28.841911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b58d23-6e2e-4ab1-8020-0437ef43894f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:28.854143 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1434/2857] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:47:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:47:28.858627 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5c9c49-9489-4035-9306-dff36bca417e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:28.868017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5c9c49-9489-4035-9306-dff36bca417e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:28.872507 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7297f96e-a13b-46f4-97cb-043dd6390d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:28.873089 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7297f96e-a13b-46f4-97cb-043dd6390d4a admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:28.873362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7297f96e-a13b-46f4-97cb-043dd6390d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:28.873627 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7297f96e-a13b-46f4-97cb-043dd6390d4a admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:28.874246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7297f96e-a13b-46f4-97cb-043dd6390d4a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:28.876292 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7297f96e-a13b-46f4-97cb-043dd6390d4a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:28.903088 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1424/2858] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:47:28 2022] POST /identity/v3/projects => generated 367 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:28.907777 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b58d23-6e2e-4ab1-8020-0437ef43894f admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:28.917751 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b58d23-6e2e-4ab1-8020-0437ef43894f admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:28.944006 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:28.977442 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4702d671-e3fa-483d-84a8-cddd33787907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:28.978048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4702d671-e3fa-483d-84a8-cddd33787907 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:28.978327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4702d671-e3fa-483d-84a8-cddd33787907 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:28.978600 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4702d671-e3fa-483d-84a8-cddd33787907 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:28.979214 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4702d671-e3fa-483d-84a8-cddd33787907 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:28.981167 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4702d671-e3fa-483d-84a8-cddd33787907 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.001960 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1435/2859] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:47:28 2022] POST /identity/v3/users => generated 284 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:29.006429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7297f96e-a13b-46f4-97cb-043dd6390d4a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.016513 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7297f96e-a13b-46f4-97cb-043dd6390d4a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.022046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d8623-9395-4951-a141-fcd72473bfe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.022634 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d8623-9395-4951-a141-fcd72473bfe8 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.022908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d8623-9395-4951-a141-fcd72473bfe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.023175 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d8623-9395-4951-a141-fcd72473bfe8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.023811 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3d8623-9395-4951-a141-fcd72473bfe8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:29.025477 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3d8623-9395-4951-a141-fcd72473bfe8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.035489 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1425/2860] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:29.039693 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4702d671-e3fa-483d-84a8-cddd33787907 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.050661 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4702d671-e3fa-483d-84a8-cddd33787907 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.054843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5cff68-085f-47d5-a1ae-e82467bb6ab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.055443 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5cff68-085f-47d5-a1ae-e82467bb6ab0 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.055720 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5cff68-085f-47d5-a1ae-e82467bb6ab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.055990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5cff68-085f-47d5-a1ae-e82467bb6ab0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.056681 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5cff68-085f-47d5-a1ae-e82467bb6ab0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:29.058275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5cff68-085f-47d5-a1ae-e82467bb6ab0 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.069211 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1436/2861] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:29.073324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d8623-9395-4951-a141-fcd72473bfe8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.083885 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d8623-9395-4951-a141-fcd72473bfe8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.088723 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b336ce0-ae0f-412b-8951-30055c7f4e27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.089400 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b336ce0-ae0f-412b-8951-30055c7f4e27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.089683 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b336ce0-ae0f-412b-8951-30055c7f4e27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.089960 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b336ce0-ae0f-412b-8951-30055c7f4e27 admin admin] PATH_INFO: `/v3/projects/7635c05dde15418484a3ba6cd1898755/users/38c5393de7a44a01ad904484b8b2010b/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.104581 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b336ce0-ae0f-412b-8951-30055c7f4e27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7635c05dde15418484a3ba6cd1898755, user_id=38c5393de7a44a01ad904484b8b2010b, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:29.107231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b336ce0-ae0f-412b-8951-30055c7f4e27 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.119982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1426/2862] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:47:29 2022] PUT /identity/v3/projects/7635c05dde15418484a3ba6cd1898755/users/38c5393de7a44a01ad904484b8b2010b/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:29.130405 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca048ada-d3d6-4121-abc9-8b74a07903ae None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.130691 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca048ada-d3d6-4121-abc9-8b74a07903ae None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.130962 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca048ada-d3d6-4121-abc9-8b74a07903ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.159020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-ca048ada-d3d6-4121-abc9-8b74a07903ae None None] MFA Rules not processed for user `38c5393de7a44a01ad904484b8b2010b`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:29.160225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ca048ada-d3d6-4121-abc9-8b74a07903ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:29.222994 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1437/2863] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:29 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:29.227446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b336ce0-ae0f-412b-8951-30055c7f4e27 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.280452 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:29.292205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b336ce0-ae0f-412b-8951-30055c7f4e27 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.297282 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c093b1-9ba9-4796-a8bf-723a63424203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.298421 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c093b1-9ba9-4796-a8bf-723a63424203 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.298749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c093b1-9ba9-4796-a8bf-723a63424203 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.299020 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c093b1-9ba9-4796-a8bf-723a63424203 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.299661 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c093b1-9ba9-4796-a8bf-723a63424203 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:29.301616 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c093b1-9ba9-4796-a8bf-723a63424203 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.328116 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1427/2864] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:47:29 2022] POST /identity/v3/projects => generated 367 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:29.332727 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca048ada-d3d6-4121-abc9-8b74a07903ae None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.343245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca048ada-d3d6-4121-abc9-8b74a07903ae None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.400086 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe660c3a-e63c-4e13-8a84-55a8c043ada4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.400643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe660c3a-e63c-4e13-8a84-55a8c043ada4 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.400924 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe660c3a-e63c-4e13-8a84-55a8c043ada4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.401199 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe660c3a-e63c-4e13-8a84-55a8c043ada4 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.401830 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe660c3a-e63c-4e13-8a84-55a8c043ada4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:29.403782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe660c3a-e63c-4e13-8a84-55a8c043ada4 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.427003 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1438/2865] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:47:29 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:29.432779 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c093b1-9ba9-4796-a8bf-723a63424203 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.445216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c093b1-9ba9-4796-a8bf-723a63424203 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.450029 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e377e13c-6d39-4dd7-b62f-831f53c51d44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.450636 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e377e13c-6d39-4dd7-b62f-831f53c51d44 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.451028 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e377e13c-6d39-4dd7-b62f-831f53c51d44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.451182 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e377e13c-6d39-4dd7-b62f-831f53c51d44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.451819 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e377e13c-6d39-4dd7-b62f-831f53c51d44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:29.453464 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e377e13c-6d39-4dd7-b62f-831f53c51d44 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.468438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1428/2866] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:29 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:47:29.473173 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe660c3a-e63c-4e13-8a84-55a8c043ada4 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.484129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe660c3a-e63c-4e13-8a84-55a8c043ada4 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.492799 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4c5c2-d2d2-4187-8205-fb7893ccd751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.493457 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4c5c2-d2d2-4187-8205-fb7893ccd751 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.493736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4c5c2-d2d2-4187-8205-fb7893ccd751 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.494010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4c5c2-d2d2-4187-8205-fb7893ccd751 admin admin] PATH_INFO: `/v3/projects/c6b21cec4e8549fea66a36c4a98fd91d/users/76c53c8898f0487d96735864e3fb58b7/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.515271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4c5c2-d2d2-4187-8205-fb7893ccd751 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6b21cec4e8549fea66a36c4a98fd91d, user_id=76c53c8898f0487d96735864e3fb58b7, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:29.518584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4c5c2-d2d2-4187-8205-fb7893ccd751 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.531255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1439/2867] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:47:29 2022] PUT /identity/v3/projects/c6b21cec4e8549fea66a36c4a98fd91d/users/76c53c8898f0487d96735864e3fb58b7/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:29.535378 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e377e13c-6d39-4dd7-b62f-831f53c51d44 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.545752 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e377e13c-6d39-4dd7-b62f-831f53c51d44 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.604362 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445bd4fb-5d9d-4cfb-8a0f-76ac80e15a39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.605079 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445bd4fb-5d9d-4cfb-8a0f-76ac80e15a39 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.605303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445bd4fb-5d9d-4cfb-8a0f-76ac80e15a39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.605578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445bd4fb-5d9d-4cfb-8a0f-76ac80e15a39 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.606269 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445bd4fb-5d9d-4cfb-8a0f-76ac80e15a39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:29.608112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445bd4fb-5d9d-4cfb-8a0f-76ac80e15a39 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.618795 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1429/2868] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:29 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:29.623020 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4c5c2-d2d2-4187-8205-fb7893ccd751 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.629395 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:29.635554 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4c5c2-d2d2-4187-8205-fb7893ccd751 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.640597 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823b29de-5cbd-4ffc-b912-4f554b49b3ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.641246 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823b29de-5cbd-4ffc-b912-4f554b49b3ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.641524 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823b29de-5cbd-4ffc-b912-4f554b49b3ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.641903 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823b29de-5cbd-4ffc-b912-4f554b49b3ba admin admin] PATH_INFO: `/v3/projects/c6b21cec4e8549fea66a36c4a98fd91d/users/76c53c8898f0487d96735864e3fb58b7/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.643907 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823b29de-5cbd-4ffc-b912-4f554b49b3ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6b21cec4e8549fea66a36c4a98fd91d, user_id=76c53c8898f0487d96735864e3fb58b7, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:29.646816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823b29de-5cbd-4ffc-b912-4f554b49b3ba admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.657048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1440/2869] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:47:29 2022] PUT /identity/v3/projects/c6b21cec4e8549fea66a36c4a98fd91d/users/76c53c8898f0487d96735864e3fb58b7/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:29.668972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afdb78b-503d-4ac8-bdbf-b3597fd11483 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.669262 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afdb78b-503d-4ac8-bdbf-b3597fd11483 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.669535 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afdb78b-503d-4ac8-bdbf-b3597fd11483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.699392 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9afdb78b-503d-4ac8-bdbf-b3597fd11483 None None] MFA Rules not processed for user `76c53c8898f0487d96735864e3fb58b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:29.700633 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9afdb78b-503d-4ac8-bdbf-b3597fd11483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:29.766189 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1430/2870] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:29 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:29.777523 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823b29de-5cbd-4ffc-b912-4f554b49b3ba admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.779315 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-823b29de-5cbd-4ffc-b912-4f554b49b3ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:29.836893 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823b29de-5cbd-4ffc-b912-4f554b49b3ba admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.840591 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f343e948-48c8-49f4-b869-48c520388e8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.841188 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f343e948-48c8-49f4-b869-48c520388e8a service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.841471 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f343e948-48c8-49f4-b869-48c520388e8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.841738 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f343e948-48c8-49f4-b869-48c520388e8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:29.851435 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f343e948-48c8-49f4-b869-48c520388e8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:29.852880 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f343e948-48c8-49f4-b869-48c520388e8a service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:29.863047 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1441/2871] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:29 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:47:29.945462 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afdb78b-503d-4ac8-bdbf-b3597fd11483 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:29.955578 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afdb78b-503d-4ac8-bdbf-b3597fd11483 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:29.959207 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191a625a-4ba7-485d-b18a-480fc453fb3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:29.959797 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191a625a-4ba7-485d-b18a-480fc453fb3c service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:29.960076 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191a625a-4ba7-485d-b18a-480fc453fb3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:29.960366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191a625a-4ba7-485d-b18a-480fc453fb3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:30.005701 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:30.018625 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191a625a-4ba7-485d-b18a-480fc453fb3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:30.020115 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191a625a-4ba7-485d-b18a-480fc453fb3c service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:30.028978 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1431/2872] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:29 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:47:34.748496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f343e948-48c8-49f4-b869-48c520388e8a service cinder] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:34.810497 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f343e948-48c8-49f4-b869-48c520388e8a service cinder] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:34.815239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:34.815848 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:34.816203 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:34.817048 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] PATH_INFO: `/v3/users/38c5393de7a44a01ad904484b8b2010b` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:34.818163 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] RBAC: Authorizing `identity:delete_user(user_id=38c5393de7a44a01ad904484b8b2010b)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:34.820182 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:34.868240 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38c5393de7a44a01ad904484b8b2010b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:34.874411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38c5393de7a44a01ad904484b8b2010b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:34.875099 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38c5393de7a44a01ad904484b8b2010b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:34.875708 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38c5393de7a44a01ad904484b8b2010b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:34.888456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38c5393de7a44a01ad904484b8b2010b'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:34.901085 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1442/2873] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:47:34 2022] DELETE /identity/v3/users/38c5393de7a44a01ad904484b8b2010b => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:34.905552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34972bb6-c30c-4f7d-8af9-e4a63f3f688f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:34.906058 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34972bb6-c30c-4f7d-8af9-e4a63f3f688f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:34.906174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34972bb6-c30c-4f7d-8af9-e4a63f3f688f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:34.925225 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-34972bb6-c30c-4f7d-8af9-e4a63f3f688f None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:34.926493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-34972bb6-c30c-4f7d-8af9-e4a63f3f688f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:34.986746 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1432/2874] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:34.997275 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:34.998680 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:35.012332 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:35.059253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac41fd4-ea0e-4414-930e-a4632708f163 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:35.062910 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940072f3-9b1c-4eec-a7fc-68519ba8063f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:35.063510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940072f3-9b1c-4eec-a7fc-68519ba8063f service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:35.063789 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940072f3-9b1c-4eec-a7fc-68519ba8063f service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:35.064059 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940072f3-9b1c-4eec-a7fc-68519ba8063f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:35.073503 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940072f3-9b1c-4eec-a7fc-68519ba8063f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:35.074918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940072f3-9b1c-4eec-a7fc-68519ba8063f service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:35.084502 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1443/2875] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:47:35.356646 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34972bb6-c30c-4f7d-8af9-e4a63f3f688f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:35.358206 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-34972bb6-c30c-4f7d-8af9-e4a63f3f688f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:35.370774 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34972bb6-c30c-4f7d-8af9-e4a63f3f688f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:35.375370 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:35.376025 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:35.376303 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:35.376603 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] PATH_INFO: `/v3/projects/7635c05dde15418484a3ba6cd1898755` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:35.377796 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=7635c05dde15418484a3ba6cd1898755)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:35.380501 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:35.426940 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7635c05dde15418484a3ba6cd1898755', 'request_id': 'req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:35.432947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7635c05dde15418484a3ba6cd1898755', 'request_id': 'req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:35.440302 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7635c05dde15418484a3ba6cd1898755', 'request_id': 'req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:35.451261 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] The token cache is being invalidate because project 7635c05dde15418484a3ba6cd1898755 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:47:35.451550 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:35.452212 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:35.453699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1433/2876] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:47:35 2022] DELETE /identity/v3/projects/7635c05dde15418484a3ba6cd1898755 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:35.458239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940072f3-9b1c-4eec-a7fc-68519ba8063f service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:35.468033 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940072f3-9b1c-4eec-a7fc-68519ba8063f service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:35.523635 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:35.524249 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:35.524556 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:35.524831 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] PATH_INFO: `/v3/users/76c53c8898f0487d96735864e3fb58b7` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:35.525940 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=76c53c8898f0487d96735864e3fb58b7)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:35.527849 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:35.575201 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '76c53c8898f0487d96735864e3fb58b7'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:35.583083 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '76c53c8898f0487d96735864e3fb58b7'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:35.583751 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '76c53c8898f0487d96735864e3fb58b7'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:35.584386 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '76c53c8898f0487d96735864e3fb58b7'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:35.596333 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-4ce5f921-cb2b-4287-8a73-a45f97919a7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '76c53c8898f0487d96735864e3fb58b7'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:35.608986 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1444/2877] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:47:35 2022] DELETE /identity/v3/users/76c53c8898f0487d96735864e3fb58b7 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:35.880591 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:35.882024 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:35.897454 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3517b818-e8a5-48fe-a0bc-71bcb9dfa9ca admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:35.954216 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:35.954883 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:35.955149 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:35.955426 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] PATH_INFO: `/v3/projects/c6b21cec4e8549fea66a36c4a98fd91d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:35.956576 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6b21cec4e8549fea66a36c4a98fd91d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:35.959296 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:35.970742 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:36.006429 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6b21cec4e8549fea66a36c4a98fd91d', 'request_id': 'req-23ce5198-a3f3-4548-8a07-49a4302174b0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:36.012308 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6b21cec4e8549fea66a36c4a98fd91d', 'request_id': 'req-23ce5198-a3f3-4548-8a07-49a4302174b0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:36.020092 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6b21cec4e8549fea66a36c4a98fd91d', 'request_id': 'req-23ce5198-a3f3-4548-8a07-49a4302174b0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:36.030470 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] The token cache is being invalidate because project c6b21cec4e8549fea66a36c4a98fd91d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:47:36.030761 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:36.031402 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-23ce5198-a3f3-4548-8a07-49a4302174b0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:36.032899 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1434/2878] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:47:35 2022] DELETE /identity/v3/projects/c6b21cec4e8549fea66a36c4a98fd91d => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:36.038049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5099b9d9-3440-4ea9-bb42-b34d5579cfd8 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.038584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5099b9d9-3440-4ea9-bb42-b34d5579cfd8 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.038584 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5099b9d9-3440-4ea9-bb42-b34d5579cfd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.059997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-5099b9d9-3440-4ea9-bb42-b34d5579cfd8 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:36.061555 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-5099b9d9-3440-4ea9-bb42-b34d5579cfd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:36.091387 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:36.123596 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1445/2879] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:36.131924 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5fa4e3-cffb-4d96-a5cc-08359322fa3c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.132205 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5fa4e3-cffb-4d96-a5cc-08359322fa3c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.132511 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5fa4e3-cffb-4d96-a5cc-08359322fa3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.151947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2f5fa4e3-cffb-4d96-a5cc-08359322fa3c None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:36.153117 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2f5fa4e3-cffb-4d96-a5cc-08359322fa3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:36.160910 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1435/2880] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:36.165035 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5099b9d9-3440-4ea9-bb42-b34d5579cfd8 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.174686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5099b9d9-3440-4ea9-bb42-b34d5579cfd8 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.179881 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5e42ae-89ac-46bb-bf58-af58483a7f1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.180515 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5e42ae-89ac-46bb-bf58-af58483a7f1f admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.180796 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5e42ae-89ac-46bb-bf58-af58483a7f1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.181068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5e42ae-89ac-46bb-bf58-af58483a7f1f admin admin] PATH_INFO: `/v3/domains` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.181713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5e42ae-89ac-46bb-bf58-af58483a7f1f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81422) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:47:36.182014 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5e42ae-89ac-46bb-bf58-af58483a7f1f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.183592 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5e42ae-89ac-46bb-bf58-af58483a7f1f admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.199909 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1446/2881] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:47:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:47:36.204112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5fa4e3-cffb-4d96-a5cc-08359322fa3c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.213360 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5fa4e3-cffb-4d96-a5cc-08359322fa3c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.217992 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58daa2e2-a971-4fee-8923-1a540cb788a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.218574 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58daa2e2-a971-4fee-8923-1a540cb788a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.218850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58daa2e2-a971-4fee-8923-1a540cb788a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.219118 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58daa2e2-a971-4fee-8923-1a540cb788a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.219741 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58daa2e2-a971-4fee-8923-1a540cb788a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.221668 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58daa2e2-a971-4fee-8923-1a540cb788a7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.249420 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1436/2882] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:47:36 2022] POST /identity/v3/projects => generated 379 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:36.254276 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5e42ae-89ac-46bb-bf58-af58483a7f1f admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.264289 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5e42ae-89ac-46bb-bf58-af58483a7f1f admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.320131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b4b09e-2e3e-4c53-ace6-f16ea60236f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.320762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b4b09e-2e3e-4c53-ace6-f16ea60236f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.321043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b4b09e-2e3e-4c53-ace6-f16ea60236f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.321327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b4b09e-2e3e-4c53-ace6-f16ea60236f7 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.321934 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b4b09e-2e3e-4c53-ace6-f16ea60236f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.323845 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b4b09e-2e3e-4c53-ace6-f16ea60236f7 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.344496 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1447/2883] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:47:36 2022] POST /identity/v3/users => generated 290 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:36.349119 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58daa2e2-a971-4fee-8923-1a540cb788a7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.359339 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58daa2e2-a971-4fee-8923-1a540cb788a7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.364098 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb14e6c-fe04-45ce-92a2-a3f93293ad40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.364726 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb14e6c-fe04-45ce-92a2-a3f93293ad40 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.365013 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb14e6c-fe04-45ce-92a2-a3f93293ad40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.365284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb14e6c-fe04-45ce-92a2-a3f93293ad40 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.365930 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb14e6c-fe04-45ce-92a2-a3f93293ad40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.367516 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb14e6c-fe04-45ce-92a2-a3f93293ad40 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.377681 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1437/2884] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:36.382171 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b4b09e-2e3e-4c53-ace6-f16ea60236f7 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.392036 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b4b09e-2e3e-4c53-ace6-f16ea60236f7 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.397323 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c858d8-2477-4659-8da0-0ccd7454a886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.397911 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c858d8-2477-4659-8da0-0ccd7454a886 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.398190 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c858d8-2477-4659-8da0-0ccd7454a886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.398464 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c858d8-2477-4659-8da0-0ccd7454a886 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.399107 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c858d8-2477-4659-8da0-0ccd7454a886 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.400740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c858d8-2477-4659-8da0-0ccd7454a886 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.411248 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1448/2885] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:36.415404 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb14e6c-fe04-45ce-92a2-a3f93293ad40 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.425245 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb14e6c-fe04-45ce-92a2-a3f93293ad40 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.430065 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b64d8c8-6aae-48c6-9b9f-bfd712b9eb51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.430694 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b64d8c8-6aae-48c6-9b9f-bfd712b9eb51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.430972 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b64d8c8-6aae-48c6-9b9f-bfd712b9eb51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.431249 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b64d8c8-6aae-48c6-9b9f-bfd712b9eb51 admin admin] PATH_INFO: `/v3/projects/777ab468dc0d4c3388645a11527f381c/users/61d2f1a910734f4b99cf8dc1efdf092a/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.446343 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b64d8c8-6aae-48c6-9b9f-bfd712b9eb51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=777ab468dc0d4c3388645a11527f381c, user_id=61d2f1a910734f4b99cf8dc1efdf092a, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.449075 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b64d8c8-6aae-48c6-9b9f-bfd712b9eb51 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.462471 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1438/2886] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:47:36 2022] PUT /identity/v3/projects/777ab468dc0d4c3388645a11527f381c/users/61d2f1a910734f4b99cf8dc1efdf092a/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:36.473972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa54f9a-43c8-481d-90b6-85322f8eb72e None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.474255 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa54f9a-43c8-481d-90b6-85322f8eb72e None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.474526 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa54f9a-43c8-481d-90b6-85322f8eb72e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.501412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-6aa54f9a-43c8-481d-90b6-85322f8eb72e None None] MFA Rules not processed for user `61d2f1a910734f4b99cf8dc1efdf092a`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:36.502606 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-6aa54f9a-43c8-481d-90b6-85322f8eb72e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:36.565933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1449/2887] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:36 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:36.570961 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b64d8c8-6aae-48c6-9b9f-bfd712b9eb51 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.631926 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b64d8c8-6aae-48c6-9b9f-bfd712b9eb51 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.636630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397a71d7-be4f-4f77-8aa4-089b0b291e11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.637221 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a71d7-be4f-4f77-8aa4-089b0b291e11 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.637499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a71d7-be4f-4f77-8aa4-089b0b291e11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.637773 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a71d7-be4f-4f77-8aa4-089b0b291e11 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.638408 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397a71d7-be4f-4f77-8aa4-089b0b291e11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.640315 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397a71d7-be4f-4f77-8aa4-089b0b291e11 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.667268 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1439/2888] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:47:36 2022] POST /identity/v3/projects => generated 379 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:36.671206 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa54f9a-43c8-481d-90b6-85322f8eb72e None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.681956 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa54f9a-43c8-481d-90b6-85322f8eb72e None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.708793 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:36.739835 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4a3d1-4f25-4e7e-9de3-eb78ce5c21a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.740487 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4a3d1-4f25-4e7e-9de3-eb78ce5c21a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.740775 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4a3d1-4f25-4e7e-9de3-eb78ce5c21a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.741049 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f4a3d1-4f25-4e7e-9de3-eb78ce5c21a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.741669 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4a3d1-4f25-4e7e-9de3-eb78ce5c21a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.743582 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f4a3d1-4f25-4e7e-9de3-eb78ce5c21a6 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.766808 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1450/2889] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:47:36 2022] POST /identity/v3/users => generated 296 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:36.767934 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397a71d7-be4f-4f77-8aa4-089b0b291e11 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.778708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397a71d7-be4f-4f77-8aa4-089b0b291e11 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.783361 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dbe4fd-b689-468d-93eb-049157278b5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.783951 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dbe4fd-b689-468d-93eb-049157278b5c admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.784231 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dbe4fd-b689-468d-93eb-049157278b5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.784536 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dbe4fd-b689-468d-93eb-049157278b5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.785326 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dbe4fd-b689-468d-93eb-049157278b5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.787030 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4dbe4fd-b689-468d-93eb-049157278b5c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.797447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1440/2890] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 26 01:47:36.802143 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4a3d1-4f25-4e7e-9de3-eb78ce5c21a6 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.811856 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f4a3d1-4f25-4e7e-9de3-eb78ce5c21a6 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.817282 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcff86b-ff2c-4099-ab31-263da3212f40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.817913 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcff86b-ff2c-4099-ab31-263da3212f40 admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.818188 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcff86b-ff2c-4099-ab31-263da3212f40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.818455 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcff86b-ff2c-4099-ab31-263da3212f40 admin admin] PATH_INFO: `/v3/projects/7b671c12824b4a07bc1f4a17acb65b43/users/45c88d60100b472c827ee65cf77d03b6/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.836053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcff86b-ff2c-4099-ab31-263da3212f40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b671c12824b4a07bc1f4a17acb65b43, user_id=45c88d60100b472c827ee65cf77d03b6, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.838710 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcff86b-ff2c-4099-ab31-263da3212f40 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.851741 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1451/2891] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:47:36 2022] PUT /identity/v3/projects/7b671c12824b4a07bc1f4a17acb65b43/users/45c88d60100b472c827ee65cf77d03b6/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:47:36.855676 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dbe4fd-b689-468d-93eb-049157278b5c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.865754 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dbe4fd-b689-468d-93eb-049157278b5c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.873528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:36.924322 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f89209-cff6-4138-ac0b-9ed77e98e239 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.924893 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f89209-cff6-4138-ac0b-9ed77e98e239 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.925177 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f89209-cff6-4138-ac0b-9ed77e98e239 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.925452 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f89209-cff6-4138-ac0b-9ed77e98e239 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.926138 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f89209-cff6-4138-ac0b-9ed77e98e239 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.927730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f89209-cff6-4138-ac0b-9ed77e98e239 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.937328 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1441/2892] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:36.941800 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcff86b-ff2c-4099-ab31-263da3212f40 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:36.951915 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcff86b-ff2c-4099-ab31-263da3212f40 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:36.956430 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107904a3-47fd-4304-92bd-c469395a6dcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:36.957598 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107904a3-47fd-4304-92bd-c469395a6dcb admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.957870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107904a3-47fd-4304-92bd-c469395a6dcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.958134 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107904a3-47fd-4304-92bd-c469395a6dcb admin admin] PATH_INFO: `/v3/projects/7b671c12824b4a07bc1f4a17acb65b43/users/45c88d60100b472c827ee65cf77d03b6/roles/3d9e246c2bd145e0a09d90666bad9c59` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:36.960004 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107904a3-47fd-4304-92bd-c469395a6dcb admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b671c12824b4a07bc1f4a17acb65b43, user_id=45c88d60100b472c827ee65cf77d03b6, role_id=3d9e246c2bd145e0a09d90666bad9c59)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:36.962633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107904a3-47fd-4304-92bd-c469395a6dcb admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:36.972330 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1452/2893] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:47:36 2022] PUT /identity/v3/projects/7b671c12824b4a07bc1f4a17acb65b43/users/45c88d60100b472c827ee65cf77d03b6/roles/3d9e246c2bd145e0a09d90666bad9c59 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:36.983048 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1b2762-70d9-448a-b8c0-91c0ba1b684a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:36.983526 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1b2762-70d9-448a-b8c0-91c0ba1b684a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:36.983630 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1b2762-70d9-448a-b8c0-91c0ba1b684a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:37.010873 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ef1b2762-70d9-448a-b8c0-91c0ba1b684a None None] MFA Rules not processed for user `45c88d60100b472c827ee65cf77d03b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:37.012082 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ef1b2762-70d9-448a-b8c0-91c0ba1b684a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:37.082417 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1442/2894] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:36 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:37.093197 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107904a3-47fd-4304-92bd-c469395a6dcb admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:37.095016 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-107904a3-47fd-4304-92bd-c469395a6dcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:37.152972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107904a3-47fd-4304-92bd-c469395a6dcb admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:37.156655 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975acd6d-a3ec-4127-9f33-db6d431b172b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:37.157242 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975acd6d-a3ec-4127-9f33-db6d431b172b service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:37.157520 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975acd6d-a3ec-4127-9f33-db6d431b172b service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:37.157792 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975acd6d-a3ec-4127-9f33-db6d431b172b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:37.166659 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975acd6d-a3ec-4127-9f33-db6d431b172b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:37.168077 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975acd6d-a3ec-4127-9f33-db6d431b172b service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:37.177643 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1453/2895] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:37 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:47:37.545514 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1b2762-70d9-448a-b8c0-91c0ba1b684a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:37.609242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1b2762-70d9-448a-b8c0-91c0ba1b684a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:37.613908 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:37.614524 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:37.614800 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:37.615070 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] PATH_INFO: `/v3/users/61d2f1a910734f4b99cf8dc1efdf092a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:37.616172 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=61d2f1a910734f4b99cf8dc1efdf092a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:37.618112 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:37.655662 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:37.666125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61d2f1a910734f4b99cf8dc1efdf092a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:37.673176 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61d2f1a910734f4b99cf8dc1efdf092a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:37.673777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61d2f1a910734f4b99cf8dc1efdf092a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:37.674374 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61d2f1a910734f4b99cf8dc1efdf092a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:37.681493 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61d2f1a910734f4b99cf8dc1efdf092a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:37.698799 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1443/2896] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:47:37 2022] DELETE /identity/v3/users/61d2f1a910734f4b99cf8dc1efdf092a => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:37.703129 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ca8289-bb8a-4302-b10a-b366be7a8293 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:37.703411 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ca8289-bb8a-4302-b10a-b366be7a8293 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:37.703684 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ca8289-bb8a-4302-b10a-b366be7a8293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:37.722567 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-99ca8289-bb8a-4302-b10a-b366be7a8293 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:37.723762 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-99ca8289-bb8a-4302-b10a-b366be7a8293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:37.780699 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1454/2897] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:37.790286 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:37.791698 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:37.851107 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea2a56f-b7bd-4fe6-9583-325f812cfe7f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:37.854730 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ece3dd-92c3-49db-8277-0203efcd2c45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:37.855325 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ece3dd-92c3-49db-8277-0203efcd2c45 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:37.855627 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ece3dd-92c3-49db-8277-0203efcd2c45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:37.855913 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ece3dd-92c3-49db-8277-0203efcd2c45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:37.865528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ece3dd-92c3-49db-8277-0203efcd2c45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:37.866947 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ece3dd-92c3-49db-8277-0203efcd2c45 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:37.876130 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1444/2898] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:47:38.142701 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ca8289-bb8a-4302-b10a-b366be7a8293 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:38.144265 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-99ca8289-bb8a-4302-b10a-b366be7a8293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:38.157692 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ca8289-bb8a-4302-b10a-b366be7a8293 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:38.162822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:38.163456 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:38.163736 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:38.164010 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] PATH_INFO: `/v3/projects/777ab468dc0d4c3388645a11527f381c` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:38.165170 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] RBAC: Authorizing `identity:delete_project(project_id=777ab468dc0d4c3388645a11527f381c)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:38.167082 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:38.189245 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:38.215595 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '777ab468dc0d4c3388645a11527f381c', 'request_id': 'req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.221809 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '777ab468dc0d4c3388645a11527f381c', 'request_id': 'req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.228135 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '777ab468dc0d4c3388645a11527f381c', 'request_id': 'req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.240573 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] The token cache is being invalidate because project 777ab468dc0d4c3388645a11527f381c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:47:38.240868 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.241522 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.242997 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1455/2899] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:47:38 2022] DELETE /identity/v3/projects/777ab468dc0d4c3388645a11527f381c => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:38.247456 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ece3dd-92c3-49db-8277-0203efcd2c45 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:38.257640 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ece3dd-92c3-49db-8277-0203efcd2c45 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:38.313368 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:38.313989 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:38.314276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:38.314820 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] PATH_INFO: `/v3/users/45c88d60100b472c827ee65cf77d03b6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:38.315967 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] RBAC: Authorizing `identity:delete_user(user_id=45c88d60100b472c827ee65cf77d03b6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:38.318000 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:38.363939 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '45c88d60100b472c827ee65cf77d03b6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.371197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '45c88d60100b472c827ee65cf77d03b6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.371813 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '45c88d60100b472c827ee65cf77d03b6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.372447 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '45c88d60100b472c827ee65cf77d03b6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.374450 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:38.381599 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-41727962-f7da-4060-9977-d27381d5f282 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '45c88d60100b472c827ee65cf77d03b6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.399463 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1445/2900] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:47:38 2022] DELETE /identity/v3/users/45c88d60100b472c827ee65cf77d03b6 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:38.677529 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:38.680296 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:38.694472 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceae349d-9d2e-4ba4-9bcd-22e86f1d111c admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:38.755234 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:38.755782 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:38.756208 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:38.756733 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] PATH_INFO: `/v3/projects/7b671c12824b4a07bc1f4a17acb65b43` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:38.757905 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b671c12824b4a07bc1f4a17acb65b43)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:38.759786 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:38.812740 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b671c12824b4a07bc1f4a17acb65b43', 'request_id': 'req-a7091bef-0927-47dd-a7ec-9bfbca0665ea'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.824709 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b671c12824b4a07bc1f4a17acb65b43', 'request_id': 'req-a7091bef-0927-47dd-a7ec-9bfbca0665ea'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.830933 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b671c12824b4a07bc1f4a17acb65b43', 'request_id': 'req-a7091bef-0927-47dd-a7ec-9bfbca0665ea'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.849314 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] The token cache is being invalidate because project 7b671c12824b4a07bc1f4a17acb65b43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:47:38.849599 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.850254 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-a7091bef-0927-47dd-a7ec-9bfbca0665ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:38.851737 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1456/2901] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:47:38 2022] DELETE /identity/v3/projects/7b671c12824b4a07bc1f4a17acb65b43 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:47:38.857063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a331cf6-d1bd-4fdf-9a4e-0e2b0d99b2fa None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:38.857438 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a331cf6-d1bd-4fdf-9a4e-0e2b0d99b2fa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:38.857711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a331cf6-d1bd-4fdf-9a4e-0e2b0d99b2fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:38.882925 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7a331cf6-d1bd-4fdf-9a4e-0e2b0d99b2fa None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:38.884183 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7a331cf6-d1bd-4fdf-9a4e-0e2b0d99b2fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:38.968556 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1446/2902] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:38.977935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c85042d-e8f6-46c5-809e-1f1f2e154eb2 None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:38.978215 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c85042d-e8f6-46c5-809e-1f1f2e154eb2 None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:38.978485 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c85042d-e8f6-46c5-809e-1f1f2e154eb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.000479 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-2c85042d-e8f6-46c5-809e-1f1f2e154eb2 None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:39.001695 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-2c85042d-e8f6-46c5-809e-1f1f2e154eb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:39.011931 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1457/2903] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:39.016236 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a331cf6-d1bd-4fdf-9a4e-0e2b0d99b2fa None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:39.030224 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a331cf6-d1bd-4fdf-9a4e-0e2b0d99b2fa None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:39.034826 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7244759c-153b-485f-940d-42bb6a488fd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:39.035532 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7244759c-153b-485f-940d-42bb6a488fd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.035814 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7244759c-153b-485f-940d-42bb6a488fd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.036095 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7244759c-153b-485f-940d-42bb6a488fd2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.037331 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7244759c-153b-485f-940d-42bb6a488fd2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 26 01:47:39.037693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7244759c-153b-485f-940d-42bb6a488fd2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:39.039305 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7244759c-153b-485f-940d-42bb6a488fd2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:39.053167 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1447/2904] 10.222.0.25 () {58 vars in 1153 bytes} [Mon Sep 26 01:47:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 26 01:47:39.058075 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c85042d-e8f6-46c5-809e-1f1f2e154eb2 None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:39.071057 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c85042d-e8f6-46c5-809e-1f1f2e154eb2 None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:39.075739 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb46015a-c1f8-41f6-8be9-935589321e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:39.076363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46015a-c1f8-41f6-8be9-935589321e1d admin admin] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.076660 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46015a-c1f8-41f6-8be9-935589321e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.076935 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46015a-c1f8-41f6-8be9-935589321e1d admin admin] PATH_INFO: `/v3/projects` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.077608 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb46015a-c1f8-41f6-8be9-935589321e1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:39.080805 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb46015a-c1f8-41f6-8be9-935589321e1d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:39.112253 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1458/2905] 10.222.0.25 () {60 vars in 1153 bytes} [Mon Sep 26 01:47:39 2022] POST /identity/v3/projects => generated 361 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:39.117580 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7244759c-153b-485f-940d-42bb6a488fd2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:39.130365 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7244759c-153b-485f-940d-42bb6a488fd2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:39.198824 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e134d86-51f0-4b5a-ada6-266aca896aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:39.199433 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e134d86-51f0-4b5a-ada6-266aca896aea admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.199719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e134d86-51f0-4b5a-ada6-266aca896aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.199996 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e134d86-51f0-4b5a-ada6-266aca896aea admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.200714 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e134d86-51f0-4b5a-ada6-266aca896aea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:39.202749 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e134d86-51f0-4b5a-ada6-266aca896aea admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:39.228140 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1448/2906] 10.222.0.25 () {60 vars in 1144 bytes} [Mon Sep 26 01:47:39 2022] POST /identity/v3/users => generated 281 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 26 01:47:39.233918 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb46015a-c1f8-41f6-8be9-935589321e1d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:39.245855 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb46015a-c1f8-41f6-8be9-935589321e1d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:39.250560 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26833e29-41ee-45cd-ac70-6bc1bde3d3b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:39.251467 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26833e29-41ee-45cd-ac70-6bc1bde3d3b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.251633 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26833e29-41ee-45cd-ac70-6bc1bde3d3b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.251885 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26833e29-41ee-45cd-ac70-6bc1bde3d3b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.252593 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26833e29-41ee-45cd-ac70-6bc1bde3d3b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:39.254839 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26833e29-41ee-45cd-ac70-6bc1bde3d3b5 admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:39.257283 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:39.266928 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1459/2907] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:39.272017 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e134d86-51f0-4b5a-ada6-266aca896aea admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:39.282956 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e134d86-51f0-4b5a-ada6-266aca896aea admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:39.288928 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1766912-e480-4b7b-951c-bf9898cda80f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:39.289523 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1766912-e480-4b7b-951c-bf9898cda80f admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.289798 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1766912-e480-4b7b-951c-bf9898cda80f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.290068 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1766912-e480-4b7b-951c-bf9898cda80f admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.290708 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1766912-e480-4b7b-951c-bf9898cda80f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:39.292476 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1766912-e480-4b7b-951c-bf9898cda80f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:39.304777 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1449/2908] 10.222.0.25 () {58 vars in 1122 bytes} [Mon Sep 26 01:47:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 26 01:47:39.310068 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26833e29-41ee-45cd-ac70-6bc1bde3d3b5 admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:39.322864 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26833e29-41ee-45cd-ac70-6bc1bde3d3b5 admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:39.328473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed78c962-774a-497b-88ba-78fbb8d5459b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:39.329118 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed78c962-774a-497b-88ba-78fbb8d5459b admin admin] REQUEST_METHOD: `PUT` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.329393 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed78c962-774a-497b-88ba-78fbb8d5459b admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.329665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed78c962-774a-497b-88ba-78fbb8d5459b admin admin] PATH_INFO: `/v3/projects/c33ba15cdf3a4206bc697f726e535bf3/users/cfcd75a86dd64ee38db9f042c7cd2832/roles/27476172dbb84c6e86b9070e02d06100` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.347305 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed78c962-774a-497b-88ba-78fbb8d5459b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c33ba15cdf3a4206bc697f726e535bf3, user_id=cfcd75a86dd64ee38db9f042c7cd2832, role_id=27476172dbb84c6e86b9070e02d06100)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:39.351519 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed78c962-774a-497b-88ba-78fbb8d5459b admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:39.366019 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1460/2909] 10.222.0.25 () {60 vars in 1483 bytes} [Mon Sep 26 01:47:39 2022] PUT /identity/v3/projects/c33ba15cdf3a4206bc697f726e535bf3/users/cfcd75a86dd64ee38db9f042c7cd2832/roles/27476172dbb84c6e86b9070e02d06100 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:39.376572 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597221ac-e73a-481f-b233-97a513b25608 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.376875 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597221ac-e73a-481f-b233-97a513b25608 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.377127 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597221ac-e73a-481f-b233-97a513b25608 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.409349 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-597221ac-e73a-481f-b233-97a513b25608 None None] MFA Rules not processed for user `cfcd75a86dd64ee38db9f042c7cd2832`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:39.411102 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-597221ac-e73a-481f-b233-97a513b25608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:39.441571 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:39.480720 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1450/2910] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:39 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:39.515401 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed78c962-774a-497b-88ba-78fbb8d5459b admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:39.517217 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-ed78c962-774a-497b-88ba-78fbb8d5459b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:39.575273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed78c962-774a-497b-88ba-78fbb8d5459b admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:39.579138 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21362fa-8b32-401c-a664-84753cfdd2f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2bb5f76eb08741f3a8c7a3a140e01e24', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:39.579770 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21362fa-8b32-401c-a664-84753cfdd2f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.580053 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21362fa-8b32-401c-a664-84753cfdd2f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.580327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21362fa-8b32-401c-a664-84753cfdd2f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.589043 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21362fa-8b32-401c-a664-84753cfdd2f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:39.590713 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21362fa-8b32-401c-a664-84753cfdd2f0 service cinder] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:39.600415 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1461/2911] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:39 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 26 01:47:39.621324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597221ac-e73a-481f-b233-97a513b25608 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:39.685083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597221ac-e73a-481f-b233-97a513b25608 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:39.689998 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:39.690623 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.690907 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.691246 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] PATH_INFO: `/v3/users/cfcd75a86dd64ee38db9f042c7cd2832` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.692482 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cfcd75a86dd64ee38db9f042c7cd2832)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:39.694412 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:39.741953 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cfcd75a86dd64ee38db9f042c7cd2832'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:39.749083 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cfcd75a86dd64ee38db9f042c7cd2832'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:39.749693 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cfcd75a86dd64ee38db9f042c7cd2832'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:39.750285 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cfcd75a86dd64ee38db9f042c7cd2832'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:39.758522 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cfcd75a86dd64ee38db9f042c7cd2832'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:39.775949 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1451/2912] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:47:39 2022] DELETE /identity/v3/users/cfcd75a86dd64ee38db9f042c7cd2832 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 26 01:47:39.780412 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cdc43f-b1e5-477b-afcf-2ee01b400fba None None] REQUEST_METHOD: `POST` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.780702 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cdc43f-b1e5-477b-afcf-2ee01b400fba None None] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.780982 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cdc43f-b1e5-477b-afcf-2ee01b400fba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.802023 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.auth.core [None req-65cdc43f-b1e5-477b-afcf-2ee01b400fba None None] MFA Rules not processed for user `d47510704d9d4183ab8e8ff6578a4a83`. Rule list: `[]` (Enabled: `True`). {{(pid=81422) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 26 01:47:39.803263 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-65cdc43f-b1e5-477b-afcf-2ee01b400fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:39.864568 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1462/2913] 10.222.0.25 () {58 vars in 958 bytes} [Mon Sep 26 01:47:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 26 01:47:39.874174 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:39.875573 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:39.931242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:39.935466 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359f916d-392f-400b-bcac-699fb1724199 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:39.940484 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9429e-5eae-4812-9f6a-0763c7a0628f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:39.941252 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9429e-5eae-4812-9f6a-0763c7a0628f service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:39.941853 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9429e-5eae-4812-9f6a-0763c7a0628f service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:39.942185 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9429e-5eae-4812-9f6a-0763c7a0628f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:39.951382 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a9429e-5eae-4812-9f6a-0763c7a0628f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:39.952848 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a9429e-5eae-4812-9f6a-0763c7a0628f service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:39.962935 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1452/2914] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:47:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:47:40.235658 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cdc43f-b1e5-477b-afcf-2ee01b400fba None None] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:47:40.237136 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-65cdc43f-b1e5-477b-afcf-2ee01b400fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:47:40.250392 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cdc43f-b1e5-477b-afcf-2ee01b400fba None None] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:47:40.256318 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:47:40.256985 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:47:40.257273 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:47:40.257535 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] PATH_INFO: `/v3/projects/c33ba15cdf3a4206bc697f726e535bf3` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:47:40.258686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=c33ba15cdf3a4206bc697f726e535bf3)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:47:40.260744 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:47:40.306705 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c33ba15cdf3a4206bc697f726e535bf3', 'request_id': 'req-91d6e024-7dc2-4136-8e93-191c69b334eb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:40.312719 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c33ba15cdf3a4206bc697f726e535bf3', 'request_id': 'req-91d6e024-7dc2-4136-8e93-191c69b334eb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:40.319055 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c33ba15cdf3a4206bc697f726e535bf3', 'request_id': 'req-91d6e024-7dc2-4136-8e93-191c69b334eb'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:40.327510 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:47:40.333532 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] The token cache is being invalidate because project c33ba15cdf3a4206bc697f726e535bf3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81422) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:47:40.333990 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:40.335196 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:47:40.336877 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1463/2915] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:47:40 2022] DELETE /identity/v3/projects/c33ba15cdf3a4206bc697f726e535bf3 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:47:49.959336 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:48:03.449162 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9429e-5eae-4812-9f6a-0763c7a0628f service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:48:03.450783 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b2a9429e-5eae-4812-9f6a-0763c7a0628f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:48:03.526125 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9429e-5eae-4812-9f6a-0763c7a0628f service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:48:03.591230 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62456c7-9e31-468c-85bc-906c5551d28a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '381ad7c239f444da90369dc58779b86c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['admin', 'ResellerAdmin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:48:03.591833 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62456c7-9e31-468c-85bc-906c5551d28a service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:48:03.592113 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62456c7-9e31-468c-85bc-906c5551d28a service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:48:03.592423 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62456c7-9e31-468c-85bc-906c5551d28a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:48:03.594337 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f62456c7-9e31-468c-85bc-906c5551d28a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:48:03.607366 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62456c7-9e31-468c-85bc-906c5551d28a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:48:03.608980 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62456c7-9e31-468c-85bc-906c5551d28a service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:48:03.620500 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1453/2916] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:48:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:48:13.303968 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:48:13.305377 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:48:13.362735 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d6e024-7dc2-4136-8e93-191c69b334eb admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:48:13.366534 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242d81f-32a8-4ada-be0a-21c902e4015c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0478ebab37fb4f488eef84f7a241e206', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:48:13.367132 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242d81f-32a8-4ada-be0a-21c902e4015c service neutron] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:48:13.367410 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242d81f-32a8-4ada-be0a-21c902e4015c service neutron] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:48:13.367683 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242d81f-32a8-4ada-be0a-21c902e4015c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:48:13.369438 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d242d81f-32a8-4ada-be0a-21c902e4015c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:48:13.431374 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242d81f-32a8-4ada-be0a-21c902e4015c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:48:13.432822 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242d81f-32a8-4ada-be0a-21c902e4015c service neutron] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:48:13.442666 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1464/2917] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:48:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:48:13.616575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=62 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:48:16.551278 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62456c7-9e31-468c-85bc-906c5551d28a service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:48:16.554734 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f62456c7-9e31-468c-85bc-906c5551d28a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:48:16.675711 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62456c7-9e31-468c-85bc-906c5551d28a service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:48:16.681792 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b104891-c594-4eb4-b7d5-09203aba8466 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:48:16.681919 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b104891-c594-4eb4-b7d5-09203aba8466 service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:48:16.682197 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b104891-c594-4eb4-b7d5-09203aba8466 service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:48:16.682445 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b104891-c594-4eb4-b7d5-09203aba8466 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:48:16.684528 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2b104891-c594-4eb4-b7d5-09203aba8466 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:48:16.705085 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b104891-c594-4eb4-b7d5-09203aba8466 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:48:16.706758 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b104891-c594-4eb4-b7d5-09203aba8466 service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:48:16.720046 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1454/2918] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:48:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:48:19.775271 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242d81f-32a8-4ada-be0a-21c902e4015c service neutron] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:48:19.776846 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-d242d81f-32a8-4ada-be0a-21c902e4015c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:48:19.790185 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242d81f-32a8-4ada-be0a-21c902e4015c service neutron] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:48:19.794843 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:48:19.795473 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:48:19.795804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:48:19.796026 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] PATH_INFO: `/v3/users/a47ec612bce54bc8b2062b03d36d10d4` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:48:19.797448 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] RBAC: Authorizing `identity:delete_user(user_id=a47ec612bce54bc8b2062b03d36d10d4)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:48:19.799363 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:48:19.846627 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a47ec612bce54bc8b2062b03d36d10d4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:19.854225 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a47ec612bce54bc8b2062b03d36d10d4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:19.854859 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a47ec612bce54bc8b2062b03d36d10d4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:19.855450 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a47ec612bce54bc8b2062b03d36d10d4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:19.867946 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a47ec612bce54bc8b2062b03d36d10d4'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:19.879904 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1465/2919] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:48:19 2022] DELETE /identity/v3/users/a47ec612bce54bc8b2062b03d36d10d4 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:48:20.056658 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b104891-c594-4eb4-b7d5-09203aba8466 service placement] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:48:20.068791 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b104891-c594-4eb4-b7d5-09203aba8466 service placement] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:48:20.128324 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:48:20.128884 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:48:20.129168 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:48:20.129446 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] PATH_INFO: `/v3/projects/786ed33f027946f581147e672b947df0` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:48:20.130575 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=786ed33f027946f581147e672b947df0)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:48:20.132718 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:48:20.179039 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '786ed33f027946f581147e672b947df0', 'request_id': 'req-379bf363-b311-4f5f-ab3d-49a84f5e0a59'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.185276 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '786ed33f027946f581147e672b947df0', 'request_id': 'req-379bf363-b311-4f5f-ab3d-49a84f5e0a59'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.194049 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '786ed33f027946f581147e672b947df0', 'request_id': 'req-379bf363-b311-4f5f-ab3d-49a84f5e0a59'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.206242 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] The token cache is being invalidate because project 786ed33f027946f581147e672b947df0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:48:20.206533 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.207196 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.208719 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1455/2920] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:48:20 2022] DELETE /identity/v3/projects/786ed33f027946f581147e672b947df0 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:48:20.213325 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:48:20.214870 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:48:20.228686 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebb1bc4-eb3b-4d97-81a1-21792e8de44d admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:48:20.246239 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:48:20.286189 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:48:20.286813 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:48:20.287219 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:48:20.287390 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] PATH_INFO: `/v3/users/848cc8f602ad47189c95c80b307f6700` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:48:20.289191 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] RBAC: Authorizing `identity:delete_user(user_id=848cc8f602ad47189c95c80b307f6700)` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:48:20.291113 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:48:20.335514 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '848cc8f602ad47189c95c80b307f6700'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.342804 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '848cc8f602ad47189c95c80b307f6700'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.343414 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '848cc8f602ad47189c95c80b307f6700'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.344080 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '848cc8f602ad47189c95c80b307f6700'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.356611 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.notifications [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '848cc8f602ad47189c95c80b307f6700'} {{(pid=81422) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.368972 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1466/2921] 10.222.0.25 () {58 vars in 1224 bytes} [Mon Sep 26 01:48:20 2022] DELETE /identity/v3/users/848cc8f602ad47189c95c80b307f6700 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:48:20.538891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:48:20.540485 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:48:20.554382 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379bf363-b311-4f5f-ab3d-49a84f5e0a59 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:48:20.571680 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:48:20.612416 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd47510704d9d4183ab8e8ff6578a4a83', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7124d36dd410484faf5f0034e732bac5', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:48:20.613035 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:48:20.613314 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:48:20.613588 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] PATH_INFO: `/v3/projects/fd28bfe672a742128e64c0b0badce537` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:48:20.614699 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd28bfe672a742128e64c0b0badce537)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:48:20.616891 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:48:20.662849 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd28bfe672a742128e64c0b0badce537', 'request_id': 'req-3342d359-2c89-4b07-90d1-8b049ad1929c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.669063 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd28bfe672a742128e64c0b0badce537', 'request_id': 'req-3342d359-2c89-4b07-90d1-8b049ad1929c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.676982 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd28bfe672a742128e64c0b0badce537', 'request_id': 'req-3342d359-2c89-4b07-90d1-8b049ad1929c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.688552 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] The token cache is being invalidate because project fd28bfe672a742128e64c0b0badce537 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 26 01:48:20.688850 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.689499 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3342d359-2c89-4b07-90d1-8b049ad1929c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 26 01:48:20.690990 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1456/2922] 10.222.0.25 () {58 vars in 1233 bytes} [Mon Sep 26 01:48:20 2022] DELETE /identity/v3/projects/fd28bfe672a742128e64c0b0badce537 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 26 01:48:30.365131 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=43,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:48:30.685284 dv-cinder-859203-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 26 01:50:17.898342 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] Authenticating user token {{(pid=81422) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 26 01:50:17.899816 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:50:17.969862 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6e27e1-4570-48e9-b100-8ae21edfddcf admin admin] Validating token access rules against request {{(pid=81422) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 26 01:50:18.028009 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe87442-91ef-469e-a6d5-d812e598394a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '427016f8da514dc4bbed21de08207dbb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '205f567457bc42f9b1a00721cfd6c41f', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81422) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 26 01:50:18.028637 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe87442-91ef-469e-a6d5-d812e598394a service placement] REQUEST_METHOD: `GET` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 26 01:50:18.028914 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe87442-91ef-469e-a6d5-d812e598394a service placement] SCRIPT_NAME: `/identity` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 26 01:50:18.029184 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe87442-91ef-469e-a6d5-d812e598394a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81422) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 26 01:50:18.030922 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.fernet_utils [None req-8fe87442-91ef-469e-a6d5-d812e598394a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81422) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 26 01:50:18.042243 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe87442-91ef-469e-a6d5-d812e598394a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 26 01:50:18.044665 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe87442-91ef-469e-a6d5-d812e598394a service placement] RBAC: Authorization granted {{(pid=81422) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 26 01:50:18.055327 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: [pid: 81422|app: 0|req: 1467/2923] 10.222.0.25 () {60 vars in 1365 bytes} [Mon Sep 26 01:50:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 26 01:50:28.050416 dv-cinder-859203-2-iscsi devstack@keystone.service[81422]: DEBUG dbcounter [-] [81422] Writing DB stats keystone:SELECT=58 {{(pid=81422) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}